知识点
相关文章
更多最近更新
更多redhat6.4上build storm 0.9.0.1
2019-03-02 23:45|来源: 网路
1.安装mvn
2.下载源代码
3.build
mvn package
过程中出现问题,clojars.org 访问不了。通过私服映射clojars.org并在pom.xml中将dependency的地址改掉。
重新运行mvn package
发现问题:
java.lang.RuntimeException: Pipe to subprocess seems to be broken! No output read.
Shell Process Exception:
/tmp/a3a2aead-499f-4f93-8390-0650f2d75d0f/supervisor/stormdist/test-1-1392348739/resources/storm.rb:19:in `require': no such file to load -- rubygems (LoadError)
from /tmp/a3a2aead-499f-4f93-8390-0650f2d75d0f/supervisor/stormdist/test-1-1392348739/resources/storm.rb:19
from tester_spout.rb:19:in `require'
from tester_spout.rb:19
at backtype.storm.utils.ShellProcess.readString(ShellProcess.java:135) ~[classes/:na]
at backtype.storm.utils.ShellProcess.readMessage(ShellProcess.java:81) ~[classes/:na]
at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:62) ~[classes/:na]
at backtype.storm.spout.ShellSpout.open(ShellSpout.java:53) ~[classes/:na]
at backtype.storm.daemon.executor$fn__3103$fn__3118.invoke(executor.clj:519) ~[classes/:na]
at backtype.storm.util$async_loop$fn__291.invoke(util.clj:431) ~[classes/:na]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_31]
先把系统默认安装的ruby1.8.7卸载掉
yum erase ruby
然后在下载一个ruby1.9.3下来
./configure
make
make install
重启机器就好了
在运行storm-book上的第一二章的例子时,发现如下错误:
4237 [Thread-9-EventThread] WARN backtype.storm.cluster - Received event :disconnected::none: with disconnected Zookeeper.
4901 [Thread-9-SendThread(localhost:2000)] WARN org.apache.zookeeper.ClientCnxn - Session 0x1442eaea3c4000b for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.6.0_35]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599) ~[na:1.6.0_35]
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119) ~[zookeeper-3.3.3.jar:3.3.3-1073969]
根据错误消息中看到,客户端用的zookeeper版本是3.3.3,而集群的zookeeper版本是3.4.5.从eclipse中的pom界面可以搜到是由下列依赖传递导致的:
storm-0.9.0.1-->curator-framework-1.0.1-->curator-client-1.0.1-->zookeeper-3.3.3
将zookeeper集群将为3.3.3版本
发现还是不行。后来想到我用的第2章的例子是LocalCluster,猜想应该是运行在一个节点上,而我的集群是用4个节点搭建的。而且其中zk集群为3台机器。
于是尝试将conf/storm.yaml中的
storm.zookeeper.servers:
- jfp3-1
- jfp3-2
- jfp3-3
改为只有一条jfp3-1
并重启zk.再次启动nimbus,ui,supervisor,然后提交任务。日志如下:
[root@jfp3-1 ~]# storm jar storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar storm.starter.WordCountTopology mystorm
Running: java -client -Dstorm.options= -Dstorm.home=/opt/storm/storm-0.9.0.1 -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /opt/storm/storm-0.9.0.1/storm-core-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/storm-netty-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/storm-console-logging-0.9.0.1.jar:/opt/storm/storm-0.9.0.1/lib/junit-3.8.1.jar:/opt/storm/storm-0.9.0.1/lib/commons-io-1.4.jar:/opt/storm/storm-0.9.0.1/lib/minlog-1.2.jar:/opt/storm/storm-0.9.0.1/lib/core.incubator-0.1.0.jar:/opt/storm/storm-0.9.0.1/lib/jetty-util-6.1.26.jar:/opt/storm/storm-0.9.0.1/lib/ring-core-1.1.5.jar:/opt/storm/storm-0.9.0.1/lib/clj-time-0.4.1.jar:/opt/storm/storm-0.9.0.1/lib/clout-1.0.1.jar:/opt/storm/storm-0.9.0.1/lib/json-simple-1.1.jar:/opt/storm/storm-0.9.0.1/lib/curator-client-1.0.1.jar:/opt/storm/storm-0.9.0.1/lib/reflectasm-1.07-shaded.jar:/opt/storm/storm-0.9.0.1/lib/disruptor-2.10.1.jar:/opt/storm/storm-0.9.0.1/lib/commons-logging-1.1.1.jar:/opt/storm/storm-0.9.0.1/lib/logback-classic-1.0.6.jar:/opt/storm/storm-0.9.0.1/lib/jgrapht-0.8.3.jar:/opt/storm/storm-0.9.0.1/lib/logback-core-1.0.6.jar:/opt/storm/storm-0.9.0.1/lib/libthrift7-0.7.0-2.jar:/opt/storm/storm-0.9.0.1/lib/httpcore-4.1.jar:/opt/storm/storm-0.9.0.1/lib/snakeyaml-1.11.jar:/opt/storm/storm-0.9.0.1/lib/commons-codec-1.4.jar:/opt/storm/storm-0.9.0.1/lib/clojure-1.4.0.jar:/opt/storm/storm-0.9.0.1/lib/servlet-api-2.5.jar:/opt/storm/storm-0.9.0.1/lib/mockito-all-1.9.5.jar:/opt/storm/storm-0.9.0.1/lib/hiccup-0.3.6.jar:/opt/storm/storm-0.9.0.1/lib/guava-13.0.jar:/opt/storm/storm-0.9.0.1/lib/commons-exec-1.1.jar:/opt/storm/storm-0.9.0.1/lib/tools.logging-0.2.3.jar:/opt/storm/storm-0.9.0.1/lib/carbonite-1.5.0.jar:/opt/storm/storm-0.9.0.1/lib/tools.macro-0.1.0.jar:/opt/storm/storm-0.9.0.1/lib/servlet-api-2.5-20081211.jar:/opt/storm/storm-0.9.0.1/lib/log4j-over-slf4j-1.6.6.jar:/opt/storm/storm-0.9.0.1/lib/clj-stacktrace-0.2.2.jar:/opt/storm/storm-0.9.0.1/lib/jline-0.9.94.jar:/opt/storm/storm-0.9.0.1/lib/ring-jetty-adapter-0.3.11.jar:/opt/storm/storm-0.9.0.1/lib/tools.nrepl-0.2.3.jar:/opt/storm/storm-0.9.0.1/lib/asm-4.0.jar:/opt/storm/storm-0.9.0.1/lib/slf4j-api-1.6.5.jar:/opt/storm/storm-0.9.0.1/lib/commons-lang-2.5.jar:/opt/storm/storm-0.9.0.1/lib/joda-time-2.0.jar:/opt/storm/storm-0.9.0.1/lib/compojure-1.1.3.jar:/opt/storm/storm-0.9.0.1/lib/ring-devel-0.3.11.jar:/opt/storm/storm-0.9.0.1/lib/curator-framework-1.0.1.jar:/opt/storm/storm-0.9.0.1/lib/httpclient-4.1.1.jar:/opt/storm/storm-0.9.0.1/lib/netty-3.6.3.Final.jar:/opt/storm/storm-0.9.0.1/lib/commons-fileupload-1.2.1.jar:/opt/storm/storm-0.9.0.1/lib/zookeeper-3.3.3.jar:/opt/storm/storm-0.9.0.1/lib/tools.cli-0.2.2.jar:/opt/storm/storm-0.9.0.1/lib/jetty-6.1.26.jar:/opt/storm/storm-0.9.0.1/lib/kryo-2.17.jar:/opt/storm/storm-0.9.0.1/lib/clojure-complete-0.2.3.jar:/opt/storm/storm-0.9.0.1/lib/math.numeric-tower-0.0.1.jar:/opt/storm/storm-0.9.0.1/lib/ring-servlet-0.3.11.jar:/opt/storm/storm-0.9.0.1/lib/objenesis-1.2.jar:/opt/storm/storm-0.9.0.1/lib/jzmq-2.1.0.jar:storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar:/opt/storm/storm-0.9.0.1/conf:/opt/storm/storm-0.9.0.1/bin -Dstorm.jar=storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar storm.starter.WordCountTopology mystorm
344 [main] INFO backtype.storm.StormSubmitter - Jar not uploaded to master yet. Submitting jar...
357 [main] INFO backtype.storm.StormSubmitter - Uploading topology jar storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar to assigned location: /var/storm/workdir/nimbus/inbox/stormjar-c00ff052-802b-40c1-bc72-83df035c2358.jar
608 [main] INFO backtype.storm.StormSubmitter - Successfully uploaded topology jar to assigned location: /var/storm/workdir/nimbus/inbox/stormjar-c00ff052-802b-40c1-bc72-83df035c2358.jar
608 [main] INFO backtype.storm.StormSubmitter - Submitting topology mystorm in distributed mode with conf {"topology.workers":3,"topology.debug":true}
1002 [main] INFO backtype.storm.StormSubmitter - Finished submitting topology: mystorm
而且还解决了一个原来的问题,web ui访问不了。现在可以访问webui了。
运行第2章的例子:
mvn exec:java -Dexec.mainClass="TopologyMain" -Dexec.args="src/main/resources/words.txt"
转自:http://www.cnblogs.com/littlesuccess/p/3549321
相关问答
更多-
storm0.9.0.1整合kafka需要哪些jar包[2021-12-05]
答:这个是scala的编译库喝运行时库(kafka是用sbt管理依赖的),所以建议你使用sbt,会自动下载所有依赖 -
我想我会回答我自己的问题。 基于我在github.com和javadocs中看到的...我相信pyleus不支持1.0.1。 javadocs表明,在以前的版本中,IRichBolt处于backtype.storm.topology但现在它存在于org.apache.storm.topology 。 谁知道还有什么是不相容的。 所以我想现在,运行一个旧版本的Storm(我只看到0.9.4兼容的引用,所以也许0.9.6也可以)。 我确实看到有一个关于0.10.0的开放拉取请求,所以我想在支持1.0.1之前可能 ...
-
建立风暴启动者:得到错误“POM for storm-core missing”(building storm-starter: getting error “POM for storm-core missing”)[2023-03-01]
我遇到了同样的问题,并从storm-user邮件列表档案中找到了答案,请参阅 http://mail-archives.apache.org/mod_mbox/storm-user/201404.mbox/%3C72091F29-8109-4A1C-A79D-DDD2D871737B%40petrolink.com%3E I encountered the same problem, and found the answer from storm-user mailing list archives, re ... -
只要你独立运行它们,我认为不应该有任何问题。 As long as you are running them independently I don't think there should be any issues.
-
Maven正试图从https repo“ https://clojars.org/repo/ ”下载jar,但java没有客户端证书。 按照以下步骤导入客户端证书 第1步:下载客户端证书 在浏览器中打开https://clojars.org/repo/ URL(即firefox) 单击URL栏右侧的锁定图标 显示服务器URL,单击以获取右箭头,然后单击“更多信息” 弹出窗口以查看服务器的证书。 单击“查看证书”,在“详细信息”表中导出到文件CERT_FILE_NAME.crt 步骤2:将客户端证书导入mav ...
-
我想我找到了一个解决方案。 运行lein deps :tree给了我这个: Possibly confusing dependencies found: [org.apache.storm/storm-core "0.9.5"] -> [commons-codec "1.6"] overrides [com.taoensso/carmine "2.11.1"] -> [commons-codec "1.10"] Consider using these exclusions: [com.taoensso ...
-
无法使用kafka-storm向apache storm提交拓扑(Unable to submit topology to apache storm using kafka-storm)[2022-04-07]
我最终通过使用maven repo中预编译的storm-kafka版本并在拓扑中添加过滤器螺栓而不是在spout本身中进行过滤来解决这个问题。 从而消除了对storm-core和storm-kafka本地编译的jar文件的需求。 这不是一个“解决方案”,但它是解决问题的一种方法。 I eventually worked around this problem by using a pre-compiled version of storm-kafka from a maven repo and adding ... -
我终于来了! 我只是简单了,我创建了一个非常简单的redis bolt,它发布了内容,我监视了redis数据库,它正在运行。 我的工作螺栓: public class RedisBolt implements IRichBolt { protected String channel = "Somriures"; // protected String configChannel; protected OutputCollector collector ...
-
部署Storm构建JAR(Deploying Storm build JAR)[2023-11-14]
@Matthias J. Sax和大家,感谢您的帮助。 我在这里犯的错误是,我所遵循的部署过程是错误的。 要部署toplogy构建,我必须遵循以下流程, Jar必须被推入风暴AWS文件夹,然后必须在命令下运行以使其被Storm识别 rm -f * .out (nohup bin / storm nimbus> nimubus.out)& (nohup bin / storm supervisor> supervisor.out)& (nohup bin / storm jar topos / IoT.jar ... -
我昨天得到了答案,我需要的是将localhost更改为127.0.0.1,然后我在终端上启动Redis数据库,在监视器的第二个终端上启动,我的发布方法正在运行。 I came with the answer yesterday, what I needed is to change the localhost to 127.0.0.1, then I launched the Redis database on a terminal, on a second terminal the monitor, an ...