对于基于Mahout Item的CF，我们的Hadoop集群中的数据太多而机器不够(Too much data and not enough machines in our Hadoop cluster for Mahout Item-based CF)

 在工作中，我正在尝试基于Mahout的基于项目的CF包构建基于项目的推荐系统。 这是我们正在处理的问题： 
 
 用户数：6,000,000项目数：200,000首选项：10,000,000,000 
 
 如果我们的Hadoop集群中有数百台机器，我们可能会在几个小时内完成RecommenderJob。 然而，问题是因为我们是一个小型创业公司，我们的Hadoop集群在这个阶段只有大约10台机器。 理想情况下，我们希望每两天运行一次推荐工作。 
 
 为了理解问题的规模，我们在一小部分数据上应用了Mahout的基于项目的CF： 
 
 用户数：100,000项目数：80,000首选项：3,000,000 
 
 在我们的Hadoop集群上，RecommenderJob所花费的时间约为10分钟。 
 
 我的问题是，鉴于我们的硬件限制（短期内不太可能改变），我们可以做些什么来加快Mahout基于物品的CF的速度？ 

At work I'm trying to build an Item-based recommendation system based on Mahout's Item-based CF package. Here's the problem that what we are dealing with:
 
Number of users: 6,000,000 Number of items: 200,000 Preferences: 10,000,000,000
 
If we have hundreds of machines in our Hadoop cluster, we might be able to finish the RecommenderJob within several hours. However, the problem is that because we are a small startup, our Hadoop cluster has only about 10 machines at this stage. Ideally, we would like to run the recommendation job once every couple of days.
 
In order to appreciate the scale of the problem, we have applied Mahout's Item-based CF on a small subset of the data:
 
Number of users: 100,000 Number of items: 80,000 Preferences: 3,000,000
 
Time taken for the RecommenderJob is about 10 minutes on our Hadoop cluster.
 
My question is, given our hardware limitation(unlikely to change in the short term), what can we do to speed things up with Mahout's Item-based CF? 

原文：https://stackoverflow.com/questions/20794327