《Kafka+Storm+HDFS整合实践》的评论

作者：darui

darui — Wed, 14 Nov 2018 07:53:51 +0000

楼主，请问有 Strom + Druid 的例子吗

作者：你好

你好 — Fri, 24 Aug 2018 13:41:41 +0000

楼主帮忙看一下运行Kafkaspout的时候出现如下错误
21646 [Thread-17-spout-executor[3 3]] INFO o.a.c.f.i.CuratorFrameworkImpl – Starting
36729 [Thread-17-spout-executor[3 3]] ERROR o.a.c.ConnectionState – Connection timed out for connection string (192.168.52.138,192.168.52.135,192.168.52.139) and timeout (15000) / elapsed (15070)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-2.7.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:87) [curator-2.7.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115) [curator-2.7.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:492) [curator-2.7.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:214) [curator-2.7.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:1) [curator-2.7.0.jar:?]
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) [curator-2.7.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:199) [curator-2.7.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:191) [curator-2.7.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:1) [curator-2.7.0.jar:?]
at org.apache.storm.kafka.DynamicBrokersReader.getNumPartitions(DynamicBrokersReader.java:111) [storm-kafka-1.0.1.jar:1.0.1]
at org.apache.storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:84) [storm-kafka-1.0.1.jar:1.0.1]
at org.apache.storm.kafka.trident.ZkBrokerReader.(ZkBrokerReader.java:44) [storm-kafka-1.0.1.jar:1.0.1]
at org.apache.storm.kafka.KafkaUtils.makeBrokerReader(KafkaUtils.java:58) [storm-kafka-1.0.1.jar:1.0.1]
at org.apache.storm.kafka.KafkaSpout.open(KafkaSpout.java:77) [storm-kafka-1.0.1.jar:1.0.1]
at org.apache.storm.daemon.executor$fn__7885$fn__7900.invoke(executor.clj:601) [storm-core-1.0.1.jar:1.0.1]
at org.apache.storm.util$async_loop$fn__625.invoke(util.clj:482) [storm-core-1.0.1.jar:1.0.1]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

作者：nihao

nihao — Fri, 24 Aug 2018 13:40:24 +0000

楼主运行Kafkaspout的时候，报错，您看看这是为什么
21646 [Thread-17-spout-executor[3 3]] INFO o.a.c.f.i.CuratorFrameworkImpl – Starting
36729 [Thread-17-spout-executor[3 3]] ERROR o.a.c.ConnectionState – Connection timed out for connection string (192.168.52.138,192.168.52.135,192.168.52.139) and timeout (15000) / elapsed (15070)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-2.7.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:87) [curator-2.7.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115) [curator-2.7.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:492) [curator-2.7.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:214) [curator-2.7.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl$3.call(GetChildrenBuilderImpl.java:1) [curator-2.7.0.jar:?]
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) [curator-2.7.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:199) [curator-2.7.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:191) [curator-2.7.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:1) [curator-2.7.0.jar:?]
at org.apache.storm.kafka.DynamicBrokersReader.getNumPartitions(DynamicBrokersReader.java:111) [storm-kafka-1.0.1.jar:1.0.1]
at org.apache.storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:84) [storm-kafka-1.0.1.jar:1.0.1]
at org.apache.storm.kafka.trident.ZkBrokerReader.(ZkBrokerReader.java:44) [storm-kafka-1.0.1.jar:1.0.1]
at org.apache.storm.kafka.KafkaUtils.makeBrokerReader(KafkaUtils.java:58) [storm-kafka-1.0.1.jar:1.0.1]
at org.apache.storm.kafka.KafkaSpout.open(KafkaSpout.java:77) [storm-kafka-1.0.1.jar:1.0.1]
at org.apache.storm.daemon.executor$fn__7885$fn__7900.invoke(executor.clj:601) [storm-core-1.0.1.jar:1.0.1]
at org.apache.storm.util$async_loop$fn__625.invoke(util.clj:482) [storm-core-1.0.1.jar:1.0.1]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

作者：暖暖朵

暖暖朵 — Tue, 21 Aug 2018 12:11:26 +0000

博主，代码报错了，想要一份完整的 pom，急求回复啊！在线等，谢谢博主，感激不尽

作者：qixingye

qixingye — Tue, 17 Jul 2018 08:54:26 +0000

作为什么都不懂的小白，只是想问一下大博主，用消息中间件的话会不会造成消息处理的延时呢，如果不用的话，在spout接收消息时是不是还会产生额外处理，同样会增加延迟。谢谢~

作者：孙玉龙

孙玉龙 — Sun, 13 May 2018 09:21:31 +0000

storm on yarn 如何链接kafka

作者：Yanjun

Yanjun — Sun, 29 Apr 2018 14:19:58 +0000

实现批处理有很多种选择：
1、可以将Flume之前收集的原始日志，从业务服务器上批量同步到HDFS，或LOAD到HIve，然后基于HDFS上数据，选择合适的批处理计算框架进行处理，如Spark、Hive、Impala等等；
2、通过Flume的Sink直接将收集的数据写入HDFS，或LOAD到Hive，然后进行批处理；
3、从Kafka集群，将数据LOAD到HDFS，或LOAD到Hive，然后进行批处理；
4、Storm处理过程中，走两个分支，一个是实时处理，另一个就是直接存储数据到HDFS，然后进行批处理。
根据你实际的情况，选择一种即可，推荐第1种选择。

作者：Yanjun

Yanjun — Sun, 29 Apr 2018 14:14:30 +0000

这个类在hdfs对应的包里面。

作者：Yanjun

Yanjun — Sun, 29 Apr 2018 14:12:26 +0000

java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/client/HdfsDataOutputStream$SyncFlag
找不到这个类啊，看下你的project里面的依赖，是不是少了。

作者：大数据蚯蚓

大数据蚯蚓 — Thu, 12 Apr 2018 10:09:11 +0000

这个怎么批处理啊