首页
\
问答
\
hadoop上的python流mapreduce作业失败 - 缺少log4j？(python streaming mapreduce job on hadoop failed - missing log4j?)
hadoop上的python流mapreduce作业失败 - 缺少log4j？(python streaming mapreduce job on hadoop failed - missing log4j?)
我试图在hadoop 2.7.1上运行一个python wordcount，它安装在Ubuntu 15.10上，我收到一个错误：
log4j:WARN No appenders could be found for logger (org.apache.hadoop.ipc.Server). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.此外，我在终端中收到RunTimeException错误，并且信息流失败并且没有输出文件。
我找到了一些线程说可能缺少log4j.properties和log4j.xml，也是log4j.properties包含的示例，我尝试了一个例子但没有成功。 我在哪里可以找到Hadoop目录中的文件（如果我能找到它们），或者如何使用正确的配置创建它们？
mapcount和reducer for wordcount的代码是从这里获取的 ，它运行得非常好
input.txt|./mapper.py|sort|./reducer.py但是，我尝试了几次在hadoop上运行它并且它失败了。 当python文件被复制到hdfs时以及当它们在本地文件系统上时，我使用了不同的命令：这一个不起作用：
hadoop hadoop-streaming-2.7.1.jar -mapper /user/mapper.py -reducer /user/reducer.py -input/input_file.txt -output /user/output也不是这个：
hadoop hadoop-streaming-2.7.1.jar -mapper "python /user/mapper.py" -reducer "python /user/reducer.py" -input/input_file.txt -output /user/output这个工作（本地文件系统中的python文件）：
hadoop hadoop-streaming-2.7.1.jar -mapper "python /home/user_name/Documents/mapper.py" -reducer "python /home/user_name/Documents/reducer.py -input /user/input_file.txt -output /user/output所有文件都具有正确的权限。
输出 - 在标准开始之后 - 如下：
16/02/15 09:47:48 INFO mapreduce.Job: map 0% reduce 0% 16/02/15 09:48:05 INFO mapreduce.Job: Task Id : attempt_1455529218252_0001_m_000000_0, Status : FAILED Error: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:449) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) ... 17 more Caused by: java.lang.RuntimeException: configuration exception at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222) at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66) ... 22 more Caused by: java.io.IOException: Cannot run program "/user/mr/mapper.py": error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:209) ... 23 more Caused by: java.io.IOException: error=2, No such file or directory at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.<init>(UNIXProcess.java:248) at java.lang.ProcessImpl.start(ProcessImpl.java:134) at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ... 24 more还有更多，但最终输出是关于流媒体作业失败：
16/02/15 09:49:07 INFO mapreduce.Job: Counters: 13 Job Counters Failed map tasks=7 Killed map tasks=1 Launched map tasks=8 Other local map tasks=6 Data-local map tasks=2 Total time spent by all maps in occupied slots (ms)=135543 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=135543 Total vcore-seconds taken by all map tasks=135543 Total megabyte-seconds taken by all map tasks=138796032 Map-Reduce Framework CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 16/02/15 09:49:07 ERROR streaming.StreamJob: Job not successful! Streaming Command Failed!从hdfs调用时，python代码无法正常工作的原因是什么？
I tried to run a python wordcount on hadoop 2.7.1 which is installed on Ubuntu 15.10 and I got an error:
log4j:WARN No appenders could be found for logger (org.apache.hadoop.ipc.Server). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.Also I get RunTimeException error in the terminal and info that streaming failed and there's no output file.
I found a few threads saying that probably log4j.properties and log4j.xml are missing, also examples what should log4j.properties contain, I tried one example but no success. Where do I find the files in Hadoop directory (if I can find them) or how can I create them with the right configuration?
The code for mapper and reducer for wordcount is taken from here and it runs absolutely fine with
input.txt|./mapper.py|sort|./reducer.pyHowever, I tried several times to run it on hadoop and it fails. I used different commands trying both when python files are copied to hdfs and when they are on the local file system: This one did not work:
hadoop hadoop-streaming-2.7.1.jar -mapper /user/mapper.py -reducer /user/reducer.py -input/input_file.txt -output /user/outputnor this one:
hadoop hadoop-streaming-2.7.1.jar -mapper "python /user/mapper.py" -reducer "python /user/reducer.py" -input/input_file.txt -output /user/outputThis one did work (python files in the local file system):
hadoop hadoop-streaming-2.7.1.jar -mapper "python /home/user_name/Documents/mapper.py" -reducer "python /home/user_name/Documents/reducer.py -input /user/input_file.txt -output /user/outputAll the files have the right permissions.
The output - after the standard beginning - is as follows:
16/02/15 09:47:48 INFO mapreduce.Job: map 0% reduce 0% 16/02/15 09:48:05 INFO mapreduce.Job: Task Id : attempt_1455529218252_0001_m_000000_0, Status : FAILED Error: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:449) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) ... 17 more Caused by: java.lang.RuntimeException: configuration exception at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222) at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66) ... 22 more Caused by: java.io.IOException: Cannot run program "/user/mr/mapper.py": error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:209) ... 23 more Caused by: java.io.IOException: error=2, No such file or directory at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.<init>(UNIXProcess.java:248) at java.lang.ProcessImpl.start(ProcessImpl.java:134) at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ... 24 moreAnd there's a lot more but the final output is about the streaming job failed:
16/02/15 09:49:07 INFO mapreduce.Job: Counters: 13 Job Counters Failed map tasks=7 Killed map tasks=1 Launched map tasks=8 Other local map tasks=6 Data-local map tasks=2 Total time spent by all maps in occupied slots (ms)=135543 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=135543 Total vcore-seconds taken by all map tasks=135543 Total megabyte-seconds taken by all map tasks=138796032 Map-Reduce Framework CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 16/02/15 09:49:07 ERROR streaming.StreamJob: Job not successful! Streaming Command Failed!What could be the reason for the python code not working when invoked from hdfs?
原文：https://stackoverflow.com/questions/35398520
更新时间：2022-03-19 16:03
最满意答案
xLeft.textProperty().bind(fieldCanvas.layoutXProperty().asString("%.0f"));xLeft.textProperty().bind(fieldCanvas.layoutXProperty().asString("%.0f"));
相关问答
更多-
xLeft.textProperty().bind(fieldCanvas.layoutXProperty().asString("%.0f")); xLeft.textProperty().bind(fieldCanvas.layoutXProperty().asString("%.0f"));
-
从TextField的文档 ： 文本输入组件，允许用户输入单行无格式文本。 与以前版本的JavaFX不同，对多线输入的支持不能作为TextField控件的一部分使用，但这是TextArea控件的唯一目的。 使用TextArea代替TextField 。 如果您不希望用户能够编辑文本控件中的文本，您也可以考虑使用Label 。 From the documentation for TextField: Text input component that allows a user to enter a sin ...
-
考虑SimpleDoubleProperty并重写set和setValue方法： import javafx.beans.property.DoubleProperty; import javafx.beans.property.SimpleDoubleProperty; public class ClampedDoublePropertyTest { private DoubleProperty numProperty = new SimpleDoubleProperty(0) { ...
-
格式化Numberfield并使其在javafx Tableview中可排序(Formatting Numberfield and making it sortable in javafx Tableview)[2023-07-18]
我想，这是解决问题的方法。 colName.setComparator(new Comparator(){ @Override public int compare(String t, String t1) { try{ DecimalFormat df = new DecimalFormat("#,###,##0.00"); Double d1 =df.parse(t); Double d2 = df. ... -
在TableView中显示DoubleProperty最多两位小数(Showing DoubleProperty up to two decimal places in TableView)[2022-11-14]
尝试 myCol.setCellValueFactory(cellData -> Bindings.format("%.2f", cellData.getValue().getPropertyMyDouble())); Try myCol.setCellValueFactory(cellData -> Bindings.format("%.2f", cellData.getValue().getPropertyMyDouble())); -
删除使用PseudoClass添加的JavaFX TreeCell格式化(Removing JavaFX TreeCell formating added with PseudoClass)[2023-04-21]
正如评论中所述，答案本身提出的解决方案是完全有效的，但是处理它的更简洁的方法可能是利用内置的filled伪类Cell ，它自动应用于任何非空单元格： :filled:ambiguous-feature { /* css-styling of non-empty ambiguous cells */ } As stated in the comments, the solution proposed in the answer itself is perfectly valid, however ... -
JavaFX：着色TableView行并格式化单元格文本(JavaFX : Coloring TableView row and formatting the cell text)[2022-04-02]
1）覆盖updateItem()方法时的经验法则是处理所有逻辑分支的情况。 就像setText()对于项目空和非空情况一样，您需要在余额> = 0时设置颜色（即“其他”情况）。 将来，您可能会使用css选择器而不是内联样式，因此常见的方法是： if(condition) item.getStyleClass().add("css"); else item.getStyleClass().remove("css"); 2）你已经有了一个cellFactory，在那里格式化文本，不需要cell ... -
使用旧版bean在TableView中进行JavaFX日期格式化(JavaFX date formatting within a TableView using legacy beans)[2022-01-14]
根据James_D的建议，可以使用CellFactory而不是CellValueFactory来完成... public class FormattedTableCell -
if(choice.get() == 1) {......} if(choice.get() == 1) {......}
-
因为它们都是Property
实现，它应该可以正常工作： DoubleProperty d = new SimpleDoubleProperty(); IntegerProperty i = new SimpleIntegerProperty(); d.bindBidirectional(i); Since they're both implementations of Property it should just work: DoubleP ...