解析并加载到Hive / Hadoop中(Parsing and loading into Hive/Hadoop)

 我是hadoop map reduce框架的新手，我正在考虑使用hadoop map reduce来解析我的数据。 我有成千上万的大分隔文件，我正在考虑编写一个map reduce work来解析这些文件并将它们加载到hive datawarehouse中。 我在perl中编写了一个可以解析这些文件的解析器。 但我坚持用Hadoop map reduce做同样的事情 
 
 例如：我有一个像x = ay = bz = c ..... x = py = qz = s ..... x = 1 z = 2 ....等文件 
 
 现在我必须在hive表中将此文件作为列（x，y，z）加载，但我无法弄清楚是否可以继续使用它。 任何指导都非常有用。 
 
 这样做的另一个问题是有一些文件缺少字段y。 我必须在map reduce工作中包含该条件。 到目前为止，我已经尝试使用streaming.jar并将我的parser.pl作为mapper作为该jar文件的输入。 我认为这不是这样做的方法:)，但我只是想尝试，如果这样做。 另外，我想过使用Hive的加载功能，但如果我在hive表中指定regexserde，则缺少的列会产生问题。 
 
 我现在迷失了，如果有人可以指导我，我会感激:) 
 
 此致，Atul 

i am new to hadoop map reduce framework, and I am thinking of using hadoop map reduce to parse my data. I have thousands of big delimited files for which I am thinking of writing a map reduce job to parse those files and load them into hive datawarehouse. I have written a parser in perl which can parse those files. But I am stuck at doing the same with Hadoop map reduce
 
For example: I have a file like x=a y=b z=c..... x=p y=q z=s..... x=1 z=2 .... and so on
 
Now I have to load this file as columns (x,y,z) in hive table, but I am not able to figure out can I proceed with it. Any guidance with this would be really helpful.
 
Another problem in doing this is there are some files where the field y is missing. I have to include that condition in the map reduce job. So far, I have tried using streaming.jar and giving my parser.pl as mapper as input to that jar file. I think that is not the way to do it :), but I was just trying if that would work. Also, I thought of using load function of Hive, but the missing column will create problem if I will specify regexserde in hive table.
 
I am lost in this now, if any one could guide me with this I would be thankful :)
 
Regards, Atul

原文：https://stackoverflow.com/questions/6600843