Hive MapReduce作业提交失败“目标是一个目录”(Hive MapReduce Job Submission fails “Target is a directory”)

 我一直在玩Hadoop和它的姐妹项目，我一路上遇到了一些问题，但我终于遇到一个我找不到答案的问题： 
 
 我有一个存储在hdfs上的配置单元表作为制表符分隔的文本文件。 我可以在表上做一个基本的选择，但是一旦我使查询变得更复杂，hive就会把它变成一个map reduce工作，它会因为下面的堆栈跟踪而失败 
 
 
 
 13/11/29 08:31:00 ERROR security.UserGroupInformation：PriviledgedActionException as：hduser（auth：SIMPLE）cause：java.io.IOException：Target / tmp / hadoop- >> yarn / staging / hduser / .staging / job_1385633903169_0013 / libjars / lib / lib是一个目录13/11/29 08:31:00 ERROR security.UserGroupInformation：PriviledgedActionException as：hduser（auth：SIMPLE）cause：java.io.IOException：Target / tmp / hadoop-yarn / staging /hduser/.staging/job_1385633903169_0013/libjars/lib/lib是一个目录java.io.IOException：目标/tmp/hadoop-yarn/staging/hduser/.staging/job_1385633903169_0013/libjars/lib/lib是org中的目录。位于org.apache.hadoop.fs.FileUtil.copy（FileUtil。）的org.apache.hadoop.fs.FileUtil.checkDest（FileUtil.java:502）中的apache.hadoop.fs.FileUtil.checkDest（FileUtil.java:500）。 java：348）org.apache.hadoop.fs.FileUtil.copy（FileUtil.java:338）org.apache.hadoop.fs.FileUtil.copy（FileUtil.java:289）atg.apache.hadoop.mapreduce org.apache.hadoop.mapr中的.JobSubmitter.copyRemoteFiles（JobSubmitter.java:139） 位于org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles（JobSubmitter.java：300）的ordu.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal（JobSubmitter.java:387）中的educe.JobSubmitter.copyAndConfigureFiles（JobSubmitter.java:212） at org.apache.hadoop.mapreduce.Job $ 10.run（Job.java:1268）at org.apache.hadoop.mapreduce.Job $ 10.run（Job.java:1265）at java.security.AccessController.doPrivileged（Native方法）在org.apache.hadoop.mapreduce.Job.submit的org.apache.hadoop.security.UserGroupInformation.doAs（UserGroupInformation.java:1491）的javax.security.auth.Subject.doAs（Subject.java:415） （Job.java:1265）org.apache.hadoop.mapred.JobClient $ 1.run（JobClient.java:562）at java的org.apache.hadoop.mapred.JobClient $ 1.run（JobClient.java:557）。位于org.apache的org.apache.hadoop.security.UserGroupInformation.doAs（UserGroupInformation.java:1491）的javax.security.auth.Subject.doAs（Subject.java:415）上的security.AccessController.doPrivileged（Native Method）。 hadoop.mapred.JobClient.submitJobInternal （JobClient.java:557）org.apache.hadoop.mapred.JobClient.submitJob（JobClient.java:548）org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute（ExecDriver.java:425） ）org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute（MapRedTask.java:144）at org.apache.hadoop.hive.ql.exec.Task.executeTask（Task.java:151）at atg Org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential（TaskRunner.java:65）位于org.apache.hadoop的org.apache.hadoop.hive.ql.Driver.launchTask（Driver.java:1414）。 hive.ql.Driver.execute（Driver.java:1192）atg.apache.hadoop.hive.ql.Driver.runInternal（Driver.java:1020）org.apache.hadoop.hive.ql.Driver.run（ Driver.java:888）org.apache.hadoop.hive.cli.CliDriver.processLocalCmd（CliDriver.java:259）org.apache.hadoop.hive.cli.CliDriver.processCmd（CliDriver.java:216）at org位于org.apache.hadoop.hive.cli的org.apache.hadoop.hive.cli.CliDriver.executeDriver（CliDriver.java:781）上的.apache.hadoop.hive.cli.CliDriver.processLine（CliDriver.java:413） .CliDriver.run（CliDriver.java:675）at或 g.apache.hadoop.hive.cli.CliDriver.main（CliDriver.java:614）at sun.reflect.NativeMethodAccessorImpl.invoke0（Native Method）at sun.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl.java:57）at sun。在org.apache.hadoop.util.RunJar.main（RunJar.java:212）的java.lang.reflect.Method.invoke（Method.java:606）中反映.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）作业提交失败，异常'java.io.IOException（目标/tmp/hadoop-yarn/staging/hduser/.staging/job_1385633903169_0013/libjars/lib/lib是一个目录）'13/11/29 08:31:00 ERROR exec。任务：作业提交失败，异常为'java.io.IOException（目标/tmp/hadoop-yarn/staging/hduser/.staging/job_1385633903169_0013/libjars/lib/lib是目录）'java.io.IOException：Target / tmp /hadoop-yarn/staging/hduser/.staging/job_1385633903169_0013/libjars/lib/lib是org.apache.hadoop.fs上的org.apache.hadoop.fs.FileUtil.checkDest（FileUtil.java:500）中的目录。 FileUtil.checkDest（FileUtil.java ：502）org.apache.hadoop.fs.FileUtil.copy（FileUtil.java:348）org.apache.hadoop.fs.FileUtil.copy（FileUtil.java:338）org.apache.hadoop.fs。 FileUtil.copy（FileUtil.java:289）org.apache.hadoop.mapreduce.JobSubmitter.copyRemoteFiles（JobSubmitter.java:139）org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles（JobSubmitter.java:212）at org位于org.apache.hadoop.mapreduce.Job $ 10.run的org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal（JobSubmitter.java:387）中的.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles（JobSubmitter.java：300） Job.java:1268）atg.apache.hadoop.mapreduce.Job $ 10.run（Job.java:1265）at java.security.AccessController.doPrivileged（Native Method）at javax.security.auth.Subject.doAs（Subject .java：415）org.apache.hadoop.security.UserGroupInformation.doAs（UserGroupInformation.java:1491）org.apache.hadoop.mapreduce.Job.submit（Job.java:1265）atg.apache.hadoop。 mapred.JobClient $ 1.run（JobClient.java:562）at org.apache.hadoop.mapred.JobClient $ 1.run（JobClient.java:557）位于org.apache.hadoop.security.UserGroupInformation的javax.security.auth.Subject.doAs（Subject.java:415）的java.security.AccessController.doPrivileged（Native Method）。在org.apache的org.apache.hadoop.mapred.JobClient.submitJobInternal（JobClient.java：557）org.apache的org.apache.hadoop.mapred.JobClient.submitJob（JobClient.java:548）上的doAs（UserGroupInformation.java:1491） org.apache上的org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute（MapRedTask.java:144）中的.hadoop.hive.ql.exec.mr.ExecDriver.execute（ExecDriver.java:425） .hadoop.hive.ql.exec.Task.executeTask（Task.java:151）org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential（TaskRunner.java:65）org.apache.hadoop.hive位于org.apache.hadoop.hive.ql.Driver.runInternal（驱动程序）的org.apache.hadoop.hive.ql.Driver.execute（Driver.java:1192）的.ql.Driver.launchTask（Driver.java:1414） .java：1020）org.apache.hadoop.hive.ql.Driver.run（Driver.java:888）org.apache.hadoop.hive.cli.CliDriver.processLocalCmd（CliDriver.java:259）a t org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216）org.apache.hadoop.hive.cli.CliDriver.processLine（CliDriver.java:413）org.apache.hadoop.hive .cli.CliDriver.executeDriver（CliDriver.java:781）org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675）org.apache.hadoop.hive.cli.CliDriver.main（CliDriver） .java：614）在sun.reflect.NativeMethodAccessorImpl.invoke0（Native Method）at sun.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl.java:57）at sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）at java。 org.apache.hadoop.util.RunJar.main中的lang.reflect.Method.invoke（Method.java:606）（RunJar.java:212） 
 
 
 有问题的文件夹确实存在于dfs上，至少是“/ tmp / hadoop-yarn / staging”部分，无论我设置它的权限，hive或hadoop在作业提交时重置它们。 真正令人关注的部分是完整路径似乎是生成的文件夹名称，那么为什么软件有自己生成的东西有问题？ 为什么路径是目录的问题？ 那应该是什么呢？ 
 
 编辑：这是我正在使用的表和我正在尝试运行的查询：查询： select * from hive_flow_details where node_id = 100 limit 10; 
 
 表： 
 
 
 
 col_name data_type comment id bigint无 
 flow_versions_id int无 
 node_id int无 
 node_name字符串无 
 
 
 请记住，这种情况发生在我试图拥有任何类型的where子句的任何uery上，因为hive会将其转换为MR作业。 

I've been playing around with Hadoop and it's sister projects, and I've had a few problems along the way, but I've finally hit one that I can not find an answer to:
 
I have a hive table stored on hdfs as a tab-delimited text file. And I can do a basic select on the table, but as soon as I make the query a little more complicated, hive turns it into a map reduce job which fails with the following stack trace
 
 
 
13/11/29 08:31:00 ERROR security.UserGroupInformation: PriviledgedActionException as:hduser (auth:SIMPLE) cause:java.io.IOException: Target /tmp/hadoop-> > yarn/staging/hduser/.staging/job_1385633903169_0013/libjars/lib/lib is a directory 13/11/29 08:31:00 ERROR security.UserGroupInformation: PriviledgedActionException as:hduser (auth:SIMPLE) cause:java.io.IOException: Target /tmp/hadoop-yarn/staging/hduser/.staging/job_1385633903169_0013/libjars/lib/lib is a directory java.io.IOException: Target /tmp/hadoop-yarn/staging/hduser/.staging/job_1385633903169_0013/libjars/lib/lib is a directory at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:500) at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:502) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:348) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289) at org.apache.hadoop.mapreduce.JobSubmitter.copyRemoteFiles(JobSubmitter.java:139) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:212) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:300) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:387) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:425) at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:144) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Job Submission failed with exception 'java.io.IOException(Target /tmp/hadoop-yarn/staging/hduser/.staging/job_1385633903169_0013/libjars/lib/lib is a directory)' 13/11/29 08:31:00 ERROR exec.Task: Job Submission failed with exception 'java.io.IOException(Target /tmp/hadoop-yarn/staging/hduser/.staging/job_1385633903169_0013/libjars/lib/lib is a directory)' java.io.IOException: Target /tmp/hadoop-yarn/staging/hduser/.staging/job_1385633903169_0013/libjars/lib/lib is a directory at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:500) at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:502) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:348) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289) at org.apache.hadoop.mapreduce.JobSubmitter.copyRemoteFiles(JobSubmitter.java:139) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:212) at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:300) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:387) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:425) at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:144) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
 
 
The folder in question does exist on the dfs, at least the "/tmp/hadoop-yarn/staging" part, and no matter what I set it's permissions to, hive or hadoop resets them on a job submission. The really concerning part is that the full path appears to be a generated folder name, so why does the software have a problem with something that it has generated on its own? Why is it a problem that the path is a directory? And what should it rather be?
 
Edit: Here is the table I'm working with and the query I'm trying to run: Query: select * from hive_flow_details where node_id = 100 limit 10; 
 
Table:
 
 
 
col_name data_type comment id bigint None
 flow_versions_id int None
 node_id int None
 node_name string None 
 
 
Bear in mind, this happens to any uery that I attempt that has any kind of a where clause, as hive translates that into an MR job.

原文：https://stackoverflow.com/questions/20279809