在AWS EMR上对Hadoop输出使用LZO时，是否会对文件（存储在S3中）进行索引以便将来自动拆分？(When using LZO on Hadoop output on AWS EMR, does it index the files (stored on S3) for future automatic splitting?)

 我想对存储在S3上的Elastic Map Reduce作业输出使用LZO压缩，但不清楚文件是否自动编入索引，以便将来对该数据运行的作业会将文件拆分为多个任务。 
 
 例如，如果我的输出是一堆TSV数据行，在1GB LZO文件中，未来的地图作业将只创建1个任务，或类似（1GB / blockSize）任务（即文件未压缩时的行为） ，或者如果目录中有LZO索引文件）？ 
 
 编辑：如果没有自动完成，建议将输出设为LZO索引？ 在将文件上传到S3 之前进行索引编制？ 

I want to use LZO compression on my Elastic Map Reduce job's output that is being stored on S3, but it is not clear if the files are automatically indexed so that future jobs run on this data will split the files into multiple tasks. 
 
For example, if my output is a bunch of lines of TSV data, in a 1GB LZO file, will a future map job only create 1 task, or something like (1GB/blockSize) tasks (i.e. the behavior of when files were not compressed, or if there was a LZO index file in the directory)?
 
Edit: If this is not done automatically, what is recommended for getting my output to be LZO-indexed? Do the indexing before uploading the file to S3?

原文：https://stackoverflow.com/questions/13019996