使用oozie distcp-action将文件从hdfs目录复制到另一个目录(Copying files from a hdfs directory to another with oozie distcp-action)

 我的行为 
 
 start_fair_usage以状态okey结束，但test_copy返回 
 
Main class [org.apache.oozie.action.hadoop.DistcpMain], main() threw exception, null
 
 在/user/comverse/data/${1}_B我有很多不同的文件，其中一些我想复制到${NAME_NODE}/user/evkuzmin/output 。 为此，我尝试从包含路径数组的copy_files.sh传递路径到我需要的文件。 
 
  <action name="start_fair_usage">
    <shell xmlns="uri:oozie:shell-action:0.1">
      <job-tracker>${JOB_TRACKER}</job-tracker>
      <name-node>${NAME_NODE}</name-node>
      <exec>${copy_file}</exec>      
      <argument>${today_without_dash}</argument>
      <argument>${mta}</argument>
      <!-- <file>${path}#${start_fair_usage}</file> -->
      <file>${path}${copy_file}#${copy_file}</file>
      <capture-output/>
    </shell>
    <ok to="test_copy"/>
    <error to="KILL"/>
  </action>

  <action name="test_copy">
    <distcp xmlns="uri:oozie:distcp-action:0.2">
      <job-tracker>${JOB_TRACKER}</job-tracker>
      <name-node>${NAME_NODE}</name-node>
      <arg>${wf:actionData('start_fair_usage')['paths']}</arg>
      <!-- <arg>${NAME_NODE}/user/evkuzmin/input/*</arg> -->
      <arg>${NAME_NODE}/user/evkuzmin/output</arg>
    </distcp>
    <ok to="END"/>
    <error to="KILL"/>
  </action>
 
 start_fair_usage启动start_fair_usage 
 
echo ${1} 
echo ${2}

dirs=(
    /user/comverse/data/${1}_B
    )
args=()

for i in $(hadoop fs -ls "${dirs[@]}" | egrep ${2}.gz | awk -F " " '{print $8}')
do
    args+=("$i")
    echo "copy file - "${i}
done

paths=${args}
echo ${paths}

My actions
 
start_fair_usage ends with status okey, but test_copy returns
 
Main class [org.apache.oozie.action.hadoop.DistcpMain], main() threw exception, null
 
In /user/comverse/data/${1}_B I have a lot of different files, some of which I want to copy to ${NAME_NODE}/user/evkuzmin/output. For that I try to pass paths from copy_files.sh which holds an array of paths to the files I need.
 
  <action name="start_fair_usage">
    <shell xmlns="uri:oozie:shell-action:0.1">
      <job-tracker>${JOB_TRACKER}</job-tracker>
      <name-node>${NAME_NODE}</name-node>
      <exec>${copy_file}</exec>      
      <argument>${today_without_dash}</argument>
      <argument>${mta}</argument>
      <!-- <file>${path}#${start_fair_usage}</file> -->
      <file>${path}${copy_file}#${copy_file}</file>
      <capture-output/>
    </shell>
    <ok to="test_copy"/>
    <error to="KILL"/>
  </action>

  <action name="test_copy">
    <distcp xmlns="uri:oozie:distcp-action:0.2">
      <job-tracker>${JOB_TRACKER}</job-tracker>
      <name-node>${NAME_NODE}</name-node>
      <arg>${wf:actionData('start_fair_usage')['paths']}</arg>
      <!-- <arg>${NAME_NODE}/user/evkuzmin/input/*</arg> -->
      <arg>${NAME_NODE}/user/evkuzmin/output</arg>
    </distcp>
    <ok to="END"/>
    <error to="KILL"/>
  </action>
 
start_fair_usage starts copy_file.sh
 
echo ${1} 
echo ${2}

dirs=(
    /user/comverse/data/${1}_B
    )
args=()

for i in $(hadoop fs -ls "${dirs[@]}" | egrep ${2}.gz | awk -F " " '{print $8}')
do
    args+=("$i")
    echo "copy file - "${i}
done

paths=${args}
echo ${paths}

原文：https://stackoverflow.com/questions/43257926