无法使用kafka-storm向apache storm提交拓扑(Unable to submit topology to apache storm using kafka-storm)

 将我的拓扑结构提交给Apache Storm（版本0.10.0.2.4.0.0-169）时，我遇到了一个错误，它使用了来自storm-kafka的Kafka spout。 我正在使用github的最新版本的storm-kafka，并且正在运行kafka 0.9.0.2.4。 
 
 我可以使用maven下载的jar文件为storm-core和storm-kafka运行这个拓扑的相当普通版本，但是我需要对spout类进行更改，所以我必须从源代码下载并自己构建它们，它编译在我的maven中的ubuntu dev环境，但是在将拓扑提交到storm时我收到错误。 
 
 “找到了多个defaults.yaml资源。您可能正在使用拓扑jar构建Storm jar” 
 
 我假设这是因为我在我的maven依赖项中包含了storm-core（我从源代码编译的本地版本（并添加到我的本地maven资源库），因为在storm-kafka中需要它，并且从mvn中提取的版本缺少所需的类通过最新版本的storm-kafka）但是，如果我将storm-core设置为pom.xml中提供的，我在提交时会收到以下错误 - 
 
 “java.lang.class.NoClassDefFoundError：org / apache / storm / spout / scheme” 
 
 我假设是因为它试图使用一些包含缺少最新文件的storm-core版本。 
 
 我似乎陷入了一个循环，为了编译最新版本的storm-kafka，我需要最新版本的storm-core，需要从源代码自编译，但我似乎无法将其部署到服务器没有造成冲突。 
 
 我是否需要在风暴服务器上更新该jar？ 我担心的是，我最终需要更新它的依赖项等。 
 
 下面是我的pom.xml文件 
 
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.automatedintelligence.datastorm</groupId>
<artifactId>DataStorm</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>DataStorm</name>
<url>https://github.com/markmcgookin/DataStorm</url>
<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<repositories>
    <repository>
            <id>local-maven-repo</id>
            <url>file:///${project.basedir}/src/main/java/dependency/repo</url>
        </repository>
    <repository>
        <id>central</id>
        <url>http://central.maven.org/maven2/</url>
    </repository>
    <repository>
        <id>github-releases</id>
        <url>http://oss.sonatype.org/content/repositories/github-releases/</url>
    </repository>
    <repository>
        <id>mvnrepository</id>
        <url>http://mvnrepository.com/</url>
    </repository>
    <repository>
        <id>clojars.org</id>
        <url>http://clojars.org/repo</url>
    </repository>
    <repository>
        <id>twitter4j</id>
        <url>http://twitter4j.org/maven2</url>
    </repository>
</repositories>
<dependencies>
    <dependency>
        <groupId>com.googlecode.json-simple</groupId>
        <artifactId>json-simple</artifactId>
        <version>1.1</version>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>3.8.1</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-collections4</artifactId>
        <version>4.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hbase</groupId>
        <artifactId>hbase-client</artifactId>
        <version>1.0.0</version>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>org.slf4j</groupId>
                <artifactId>log4j-over-slf4j</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>                                                                                                                                       
        <groupId>org.apache.hadoop</groupId>                                                                                                           
        <artifactId>hadoop-common</artifactId>                                                                                                         
        <version>2.7.2</version>   
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
        </exclusions>                                                                                         
    </dependency> 
    <dependency>
        <groupId>storm</groupId>
        <artifactId>storm</artifactId>
        <version>0.9.0-rc2</version>
        <scope>provided</scope>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.11</artifactId>
        <version>0.9.0.0</version>
        <exclusions>
            <exclusion>
                <groupId>org.apache.zookeeper</groupId>
                <artifactId>zookeeper</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-kafka</artifactId>
        <version>2.0.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-core</artifactId>
        <version>2.0.0</version>
        <scope>provided</scope>
    </dependency>
</dependencies>
<build>
    <plugins>
        <!-- bind the maven-assembly-plugin to the package phase
        this will create a jar file without the Storm
        dependencies suitable for deployment to a cluster.-->
        <plugin>
            <artifactId>maven-assembly-plugin</artifactId>
            <configuration>
                <descriptorRefs>
                    <descriptorRef>jar-with-dependencies</descriptorRef>
                </descriptorRefs>
                <archive>
                    <manifest>
                        <mainClass/>
                    </manifest>
                </archive>
            </configuration>
            <executions>
                <execution>
                    <id>make-assembly</id>
                    <phase>package</phase>
                    <goals>
                        <goal>single</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>com.theoryinpractise</groupId>
            <artifactId>clojure-maven-plugin</artifactId>
            <version>1.3.8</version>
            <extensions>true</extensions>
            <configuration>
                <sourceDirectories>
                    <sourceDirectory>src/clj</sourceDirectory>
                </sourceDirectories>
            </configuration>
            <executions>
                <execution>
                    <id>compile</id>
                    <phase>compile</phase>
                    <goals>
                        <goal>compile</goal>
                    </goals>
                </execution>
                <execution>
                    <id>test</id>
                    <phase>test</phase>
                    <goals>
                        <goal>test</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>1.6</source>
                <target>1.6</target>
                <excludes>
                    <exclude>sl4j:log4j:log4j:jar:</exclude>
                </excludes>
            </configuration>
        </plugin>
    </plugins>
</build>
 

I am getting an error when submitting my topology to Apache Storm (Version 0.10.0.2.4.0.0-169) that utilities a Kafka spout from storm-kafka. I am using the latest version of storm-kafka from github, and am running kafka 0.9.0.2.4.
 
I can run a fairly vanilla version of this topology by using maven downloaded jar files for storm-core and storm-kafka but I needed to make changes to the spout class so I had to download from source and build them myself, it compiles on my ubuntu dev environment in maven, but I am getting an error when submitting the topology to storm.
 
"Found multiple defaults.yaml resources. You're probably building the Storm jars with your topology jar" 
 
I assume this is because I am including storm-core in my maven dependencies (a local version I compiled from source (and added to my local maven repository) because it is needed in storm-kafka and the version pulled from mvn is missing classes required by the latest version of storm-kafka) however, if I set the storm-core to provided in pom.xml I get the following error on submission -
 
"java.lang.class.NoClassDefFoundError: org/apache/storm/spout/scheme" 
 
Which I assume is because it is trying to use some included version of storm-core that lacks some up to date files. 
 
I seem to be stuck in a cycle that, to compile the latest version of storm-kafka I need the latest version of storm-core which needs to be self compiled from source, however I don't seem to be able to deploy this to the server without causing a clash.
 
Do I need to update that jar on the storm server? My concern here is that I will end up needing to update it's dependencies etc.
 
Below is my pom.xml file
 
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.automatedintelligence.datastorm</groupId>
<artifactId>DataStorm</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>DataStorm</name>
<url>https://github.com/markmcgookin/DataStorm</url>
<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<repositories>
    <repository>
            <id>local-maven-repo</id>
            <url>file:///${project.basedir}/src/main/java/dependency/repo</url>
        </repository>
    <repository>
        <id>central</id>
        <url>http://central.maven.org/maven2/</url>
    </repository>
    <repository>
        <id>github-releases</id>
        <url>http://oss.sonatype.org/content/repositories/github-releases/</url>
    </repository>
    <repository>
        <id>mvnrepository</id>
        <url>http://mvnrepository.com/</url>
    </repository>
    <repository>
        <id>clojars.org</id>
        <url>http://clojars.org/repo</url>
    </repository>
    <repository>
        <id>twitter4j</id>
        <url>http://twitter4j.org/maven2</url>
    </repository>
</repositories>
<dependencies>
    <dependency>
        <groupId>com.googlecode.json-simple</groupId>
        <artifactId>json-simple</artifactId>
        <version>1.1</version>
    </dependency>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>3.8.1</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-collections4</artifactId>
        <version>4.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hbase</groupId>
        <artifactId>hbase-client</artifactId>
        <version>1.0.0</version>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>org.slf4j</groupId>
                <artifactId>log4j-over-slf4j</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>                                                                                                                                       
        <groupId>org.apache.hadoop</groupId>                                                                                                           
        <artifactId>hadoop-common</artifactId>                                                                                                         
        <version>2.7.2</version>   
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
        </exclusions>                                                                                         
    </dependency> 
    <dependency>
        <groupId>storm</groupId>
        <artifactId>storm</artifactId>
        <version>0.9.0-rc2</version>
        <scope>provided</scope>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.11</artifactId>
        <version>0.9.0.0</version>
        <exclusions>
            <exclusion>
                <groupId>org.apache.zookeeper</groupId>
                <artifactId>zookeeper</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
            <exclusion> 
                <groupId>log4j</groupId>
                <artifactId>log4j</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-api</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-kafka</artifactId>
        <version>2.0.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.storm</groupId>
        <artifactId>storm-core</artifactId>
        <version>2.0.0</version>
        <scope>provided</scope>
    </dependency>
</dependencies>
<build>
    <plugins>
        <!-- bind the maven-assembly-plugin to the package phase
        this will create a jar file without the Storm
        dependencies suitable for deployment to a cluster.-->
        <plugin>
            <artifactId>maven-assembly-plugin</artifactId>
            <configuration>
                <descriptorRefs>
                    <descriptorRef>jar-with-dependencies</descriptorRef>
                </descriptorRefs>
                <archive>
                    <manifest>
                        <mainClass/>
                    </manifest>
                </archive>
            </configuration>
            <executions>
                <execution>
                    <id>make-assembly</id>
                    <phase>package</phase>
                    <goals>
                        <goal>single</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>com.theoryinpractise</groupId>
            <artifactId>clojure-maven-plugin</artifactId>
            <version>1.3.8</version>
            <extensions>true</extensions>
            <configuration>
                <sourceDirectories>
                    <sourceDirectory>src/clj</sourceDirectory>
                </sourceDirectories>
            </configuration>
            <executions>
                <execution>
                    <id>compile</id>
                    <phase>compile</phase>
                    <goals>
                        <goal>compile</goal>
                    </goals>
                </execution>
                <execution>
                    <id>test</id>
                    <phase>test</phase>
                    <goals>
                        <goal>test</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>1.6</source>
                <target>1.6</target>
                <excludes>
                    <exclude>sl4j:log4j:log4j:jar:</exclude>
                </excludes>
            </configuration>
        </plugin>
    </plugins>
</build>
 

原文：https://stackoverflow.com/questions/36402724