使用MAPRFS运行`hadoop fs`时如何启用其他日志记录？(How to enable additional logging when running `hadoop fs` with MAPRFS?)

 Bash和grep 
 
 循环遍历搜索文件并对每行进行grepping，将结果重定向到正确命名的文件： 
 
while read str; do grep -F "$str" infile > "$str".txt; done < search.txt
 
 infile是你的大文件。 这导致以下文件： 
 
==> Value003.txt <==
line1"value001","value002","Value003"
line3"value001","value002","Value003"
line4"value001","value002","Value003"
line5"value001","value002","Value003"

==> Value007.txt <==
line2"Value004","Value005","Value006","Value007"
line6"Value004","Value005","Value006","Value007"

==> Value009.txt <==
line7"value010","value022","Value009"
 
 请注意，这会多次处理非常大的文件，即使grep很快，使用Bash循环遍历文件也很慢，因此只有在search.txt相对较小时才可行。 
 
 AWK 
 
 要仅处理大文件一次，您可以使用awk迭代它，并且对于每一行检查是否有任何字符串匹配： 
 
#!/usr/bin/awk -f

# Read search file into array
NR == FNR {
    searchstr[$0]
    next
}

{
    # Iterate over search strings
    for (str in searchstr) {
        # Print to file if matches
        if (index($0, str)) {
            print $0 > str ".txt"
            # next  # Uncomment if only one search string can occur per line
            # close(str ".txt") # Uncomment if there are too many open files
        }
    }
}
 
 必须如下调用： 
 
awk -f script.awk search.txt infile
 
 在一个不太易读的单行版本中： 
 
awk 'NR==FNR{ss[$0];next}{for(s in ss)if(index($0,s))print$0>s".txt"}' search.txt infile
 
 请注意，某些awks对打开文件句柄1的数量有限制，而其他（GNU awk）可以管理更多但速度超过该限制 - 这取决于search.txt的大小。 如果它成为问题，我们可以在if子句中添加close(str ".txt")以在每次写入后关闭文件。 
 
 如果每行只能出现一个搜索字符串，我们可以取消注释循环中的next语句。 
 
 
 1原始的awk有15个打开文件的限制！ 

Bash and grep
 
Looping over the search file and grepping for each line, redirecting the result to the properly named file:
 
while read str; do grep -F "$str" infile > "$str".txt; done < search.txt
 
where infile is your large file. This results in the following files:
 
==> Value003.txt <==
line1"value001","value002","Value003"
line3"value001","value002","Value003"
line4"value001","value002","Value003"
line5"value001","value002","Value003"

==> Value007.txt <==
line2"Value004","Value005","Value006","Value007"
line6"Value004","Value005","Value006","Value007"

==> Value009.txt <==
line7"value010","value022","Value009"
 
Notice that this processes the very large file multiple times, and even though grep is fast, looping over a file with Bash is slow, so this is only viable if search.txt is relatively small.
 
Awk
 
To process the large file only once, you could iterate over it with awk, and for each line check if any of the strings match:
 
#!/usr/bin/awk -f

# Read search file into array
NR == FNR {
    searchstr[$0]
    next
}

{
    # Iterate over search strings
    for (str in searchstr) {
        # Print to file if matches
        if (index($0, str)) {
            print $0 > str ".txt"
            # next  # Uncomment if only one search string can occur per line
            # close(str ".txt") # Uncomment if there are too many open files
        }
    }
}
 
This has to be called as follows:
 
awk -f script.awk search.txt infile
 
In a less readable one-line version:
 
awk 'NR==FNR{ss[$0];next}{for(s in ss)if(index($0,s))print$0>s".txt"}' search.txt infile
 
Notice that some awks have a limit to the number of open filehandles1, and others (GNU awk) can manage more but slow down beyond that limit – this depends on the size of your search.txt. If it becomes a problem, we can add close(str ".txt") to the if clause to close the file after each write.
 
If only one search string can occur on each line, we can uncomment the next statement in the loop.
 
 
1 The original awk had a limit of 15 open files!