使用Map Reduce（Cloudera Hadoop 0.20.2）比较两个大小接近3GB的文本文件(Comparing using Map Reduce(Cloudera Hadoop 0.20.2) two text files of size of almost 3GB)

 我正在尝试在hadoop map / reduce中执行以下操作（用java编写，linux内核操作系统） 
 
 
 
 文本文件'rules-1'和'rules-2'（总共3GB大小）包含一些规则，每个规则由endline字符分隔，因此可以使用readLine（）函数读取文件。 
 
 
 这些文件'rules-1'和'rules-2'需要作为一个整体从hdfs导入到我的集群中的每个映射函数中，即这些文件不能跨不同的映射函数分割。 
 
 
 映射器映射函数的输入是一个名为'record'的文本文件（每行都以endline character结尾），所以从'record'文件中我们得到（key，value）对。 该文件是可拆分的，可以作为整个地图/缩小过程中使用的不同地图功能的输入。 
 
 
 需要做的是比较每个值（即来自记录文件的行）与'rules-1'和'rules-2'内的规则， 
 
 
 问题是，如果我将每行rule-1和rules-2文件仅抽出一次到静态数组列表，以便每个映射器可以共享相同的arraylint并尝试比较数组列表中的元素与记录中的每个输入值文件，我得到一个内存溢出错误，因为3GB不能一次存储在数组列表中。 
 
 另外，如果我一次只导入rules-1和rules-2文件中的几行，并将它们与每个值进行比较，则map / reduce将花费大量时间完成其工作。 
 
 你们能否给我提供任何其他替代想法？如何在没有内存溢出错误的情况下完成这项工作？ 如果我将这些文件1和文件2放入hdfs支持数据库或其他内容中，会有帮助吗？ 我真的会出来的想法。真的很感激，如果你们中的一些人可以提供给我你宝贵的建议。 

I'm trying to do the following in hadoop map/reduce( written in java, linux kernel OS)
 
 
 
Text files 'rules-1' and 'rules-2' (total 3GB in size) contains some rules, each rule are separated by endline character, so the files can be read using readLine() function.
 
 
These files 'rules-1' and 'rules-2' needs to be imported as a whole from hdfs in every map function in my cluster i.e. these file are not splittable across different map function.
 
 
Input to the mapper's map function is a text file called 'record' (each line is terminated by endline character), so from the 'record' file we get the (key, value) pair. The file is splittable and can be given as input to different map function used in the whole map/reduce process.
 
 
What needs to be done is compare each value(i.e. lines from record file) with the rules inside 'rules-1' and 'rules-2'
 
 
Problem is, if I pull out each line of rules-1 and rules-2 files to a static arraylist only once, so that each mapper can share the same arraylint and try to compare elements in the arraylist with the each input value from the record file, I get a memory overflow error, since 3GB cannot be stored at a time in the arraylist.
 
Alternatively, if I import only few lines from the rules-1 and rules-2 files at a time and compare them to each value, map/reduce is taking a lot time to finish its job.
 
Could you guys provide me any other alternative ideas how can this be done without the memory overflow error? Will it help if I put those file-1 and file-2 inside a hdfs supporting database or something? I'm going out of ideas actually.Would really appreciate if some of you guys could provide me your valuable suggestions.

原文：https://stackoverflow.com/questions/5606927