首页 \ 问答 \ Oozie Hadoop Streaming(Oozie Hadoop Streaming)

Oozie Hadoop Streaming(Oozie Hadoop Streaming)

 我正在尝试编写一个简单的地图，只有hadoop流式传输作业从hdfs读取数据并将其推送到vertica。 
 

 我写了几个shell脚本如下 
 

 load.sh 
 

  hadoop jar hadoop-streaming-2.7.3.2.5.3.0-37.jar -input $INPUT_DIR -mapper /user/oozie/adhoc_data_load/scripts/export.sh -output $OUTPUT_DIR

 

 export.sh 
 

 ./vsql -c "copy $TABLE from stdin delimiter E'\t' direct null '\\N';" -U $DBUSER -w $DBPWD -h $DBHOST -p $DBPORT

 

 工作流程： 
 

<action name="loadToVertica">
    <shell xmlns="uri:oozie:shell-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <exec>loadVertica.sh</exec>
            <argument>${STREAMING_JAR_PATH}</argument>
            <argument>${nameNode}/user/oozie/optus_adhoc_data/${exportDataDate}</argument>
            <argument>${TABLE_NAME_VERTICA}</argument>
            <argument>${dbHost}</argument>
            <argument>${dbName}</argument>
            <argument>${dbPassword}</argument>
            <argument>${dbPort}</argument>
            <argument>${nameNode}/user/oozie/optus_adhoc_data/output/${exportDataDate}</argument>
            <argument>vsql,export.sh</argument>
            <file>${nameNode}/user/oozie/adhoc_data_load/scripts/loadVertica.sh#loadVertica.sh</file>
            <file>${wfsBasePath}/libs/${STREAMING_JAR_PATH}#${STREAMING_JAR_PATH}</file>
            <file>${wfsBasePath}/config/vsql#vsql</file>
            <file>${wfsBasePath}/scripts/export.sh#export.sh</file>
            <capture-output/>
        </shell>
        <ok to="end"/>
       <error to="end"/>
    </action>

 

 使用Oozie用户运行它获得以下异常： 
 

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=yarn, access=WRITE, inode="/user/yarn/.staging":hdfs:hdfs:drwxr-xr-x
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)

 

 解： 
 

 添加工作流程： 
 

 HADOOP_USER_NAME = $ {WF：用户（）} 

I am trying to write a simple map only hadoop streaming job reading data from hdfs and pushing it to vertica.
 

I have written few shell scripts as below
 

load.sh
 

  hadoop jar hadoop-streaming-2.7.3.2.5.3.0-37.jar -input $INPUT_DIR -mapper /user/oozie/adhoc_data_load/scripts/export.sh -output $OUTPUT_DIR

 

export.sh
 

 ./vsql -c "copy $TABLE from stdin delimiter E'\t' direct null '\\N';" -U $DBUSER -w $DBPWD -h $DBHOST -p $DBPORT

 

Workflow:
 

<action name="loadToVertica">
    <shell xmlns="uri:oozie:shell-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <exec>loadVertica.sh</exec>
            <argument>${STREAMING_JAR_PATH}</argument>
            <argument>${nameNode}/user/oozie/optus_adhoc_data/${exportDataDate}</argument>
            <argument>${TABLE_NAME_VERTICA}</argument>
            <argument>${dbHost}</argument>
            <argument>${dbName}</argument>
            <argument>${dbPassword}</argument>
            <argument>${dbPort}</argument>
            <argument>${nameNode}/user/oozie/optus_adhoc_data/output/${exportDataDate}</argument>
            <argument>vsql,export.sh</argument>
            <file>${nameNode}/user/oozie/adhoc_data_load/scripts/loadVertica.sh#loadVertica.sh</file>
            <file>${wfsBasePath}/libs/${STREAMING_JAR_PATH}#${STREAMING_JAR_PATH}</file>
            <file>${wfsBasePath}/config/vsql#vsql</file>
            <file>${wfsBasePath}/scripts/export.sh#export.sh</file>
            <capture-output/>
        </shell>
        <ok to="end"/>
       <error to="end"/>
    </action>

 

Running it with Oozie user getting below exception:
 

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=yarn, access=WRITE, inode="/user/yarn/.staging":hdfs:hdfs:drwxr-xr-x
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)

 

Solution:
 

Add in workflow :
 

HADOOP_USER_NAME=${wf:user()}

原文：https://stackoverflow.com/questions/44498025
更新时间：2022-04-18 22:04

最满意答案

 我对你有一个想法，而不是将图像放在要滚动的元素的上方，如何将它放在它们下面，以相同的不透明度显示。 
 

 然后也改变你正在滚动的div的不透明度。 这会让它看起来像在下面。 
 

 你会添加： 
 

 

 
 
 
  
$(document).ready(function(){
  $(document).on('click', function(){
    $('#VCBody').prepend('<div class="vcBodyShadow">3</div>');
    $('.vcElementContainer').css('opacity','0.5');
  });
});
 
  
.vcBodyShadow{
    position:absolute;
    top:0;
    left:0;
    width:100%;
    height:100%;
    background-color: red;
    z-index:1;
    opacity: 0.5;
    font-size: 100px;
    font-weight: bold;
    padding: 50px;
}
#VCBody{
    position:relative;
    height:100%;
    width:100%;
    overflow:auto;
}

.vcElementContainer {
    display:flex;
    align-items:center;
    justify-content:space-around;    
    flex-wrap: wrap;      
    height: $height;
    width:$width;
    overflow:hidden; 
    position:relative;
    z-index:2;
    background-color:yellow;    
    min-height:100px;
    margin:20px;
}
 
  
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<html>
  <body>
    <div id="VCBody">
      <div class="vcElementContainer">1</div>
      <div class="vcElementContainer">2</div>
    </div>
  </body>
</html>
 
 
 

 


I have an idea for you, instead of having the image above the element you want to scroll on, how about putting it below them, with the same opacity. 
 

Then also change the opacity of the divs you're scrolling. This will give it the appearance of being underneath.
 

You would add:
 

 

 
 
 
  
$(document).ready(function(){
  $(document).on('click', function(){
    $('#VCBody').prepend('<div class="vcBodyShadow">3</div>');
    $('.vcElementContainer').css('opacity','0.5');
  });
});
 
  
.vcBodyShadow{
    position:absolute;
    top:0;
    left:0;
    width:100%;
    height:100%;
    background-color: red;
    z-index:1;
    opacity: 0.5;
    font-size: 100px;
    font-weight: bold;
    padding: 50px;
}
#VCBody{
    position:relative;
    height:100%;
    width:100%;
    overflow:auto;
}

.vcElementContainer {
    display:flex;
    align-items:center;
    justify-content:space-around;    
    flex-wrap: wrap;      
    height: $height;
    width:$width;
    overflow:hidden; 
    position:relative;
    z-index:2;
    background-color:yellow;    
    min-height:100px;
    margin:20px;
}
 
  
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<html>
  <body>
    <div id="VCBody">
      <div class="vcElementContainer">1</div>
      <div class="vcElementContainer">2</div>
    </div>
  </body>
</html>

相关问答

更多