首页 \ 问答 \ 用hadoop进行python流无法正常工作(python streaming with hadoop is not working)

用hadoop进行python流无法正常工作(python streaming with hadoop is not working)

 我正在尝试使用python运行hadoop流式map-reduce作业。 我一直收到此错误消息，不知道如何解决它。 
 

 这是我正在使用的命令。 
 

hdfs -jar /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.2.0.jar -input /input/ -output /out22 -mapper "python /mapper.py" -file /mapper.py -file /reducer.py -reducer "python /reducer.py"

 

 这是错误信息。 
 

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/Tool
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2570)
    at java.lang.Class.getMethod0(Class.java:2813)
    at java.lang.Class.getMethod(Class.java:1663)
    at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.Tool
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 6 more

 

 我正在做什么有什么不对吗？ 请帮忙。 

I'm trying to run hadoop streaming map-reduce job using python. i keep getting this error message with no idea how to fix it.
 

this is the command I'm using. 
 

hdfs -jar /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.2.0.jar -input /input/ -output /out22 -mapper "python /mapper.py" -file /mapper.py -file /reducer.py -reducer "python /reducer.py"

 

and here is the error message.
 

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/Tool
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2570)
    at java.lang.Class.getMethod0(Class.java:2813)
    at java.lang.Class.getMethod(Class.java:1663)
    at sun.launcher.LauncherHelper.getMainMethod(LauncherHelper.java:494)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:486)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.Tool
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 6 more

 

is there anything wrong with what I'm doing ? please help.

原文：https://stackoverflow.com/questions/26077154
更新时间：2022-10-28 11:10

最满意答案

 我想你不能在浏览器中使用fs模块。 在浏览器中有一个名为fs-web的单独的npm模块用于fs 。 使用npm install fs-web --save ，您可以通过在客户端文件中导入它来使用它，如下所示， 
 

 import * as fs from 'fs-web';


I think you can't use fs module in the browser. There is a separate npm module for fs in the browser called fs-web. After installing this using npm install fs-web --save, you can use it by importing it in the client side files like this,
 

 import * as fs from 'fs-web';

相关问答

更多