KafkaSpout 引起的 log4j 的问题

今天在测试 KafkaSpout 的时候突然冒出了 log4j 的问题，先是两行醒目的红色警告：

SLF4J: Detected both log4j-over-slf4j.jar AND slf4j-log4j12.jar on the class path, preempting StackOverflowError. 
SLF4J: See also http://www.slf4j.org/codes.html#log4jDelegationLoop for more details.

跟着 topology 就挂掉了，只留下了一堆无奈的日志：

5370 [Thread-14-newKafka] ERROR backtype.storm.util - Async loop died!
java.lang.NoClassDefFoundError: Could not initialize class org.apache.log4j.Log4jLoggerFactory
    at org.apache.log4j.Logger.getLogger(Logger.java:39) ~[log4j-over-slf4j-1.6.6.jar:1.6.6]
    at kafka.utils.Logging$class.logger(Logging.scala:24) ~[kafka_2.10-0.8.2.1.jar:na]
    at kafka.consumer.SimpleConsumer.logger$lzycompute(SimpleConsumer.scala:30) ~[kafka_2.10-0.8.2.1.jar:na]
    at kafka.consumer.SimpleConsumer.logger(SimpleConsumer.scala:30) ~[kafka_2.10-0.8.2.1.jar:na]
    at kafka.utils.Logging$class.info(Logging.scala:67) ~[kafka_2.10-0.8.2.1.jar:na]
    at kafka.consumer.SimpleConsumer.info(SimpleConsumer.scala:30) ~[kafka_2.10-0.8.2.1.jar:na]
    at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:74) ~[kafka_2.10-0.8.2.1.jar:na]
    at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:68) ~[kafka_2.10-0.8.2.1.jar:na]
    at kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:127) ~[kafka_2.10-0.8.2.1.jar:na]
    at kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79) ~[kafka_2.10-0.8.2.1.jar:na]
    at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:77) ~[storm-kafka-0.9.3.jar:0.9.3]
    at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:67) ~[storm-kafka-0.9.3.jar:0.9.3]
    at storm.kafka.PartitionManager.<init>(PartitionManager.java:83) ~[storm-kafka-0.9.3.jar:0.9.3]
    at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98) ~[storm-kafka-0.9.3.jar:0.9.3]
    at storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69) ~[storm-kafka-0.9.3.jar:0.9.3]
    at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:135) ~[storm-kafka-0.9.3.jar:0.9.3]
    at backtype.storm.daemon.executor$fn__3373$fn__3388$fn__3417.invoke(executor.clj:565) ~[storm-core-0.9.3.jar:0.9.3]
    at backtype.storm.util$async_loop$fn__464.invoke(util.clj:463) ~[storm-core-0.9.3.jar:0.9.3]
    at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
    at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]

追根溯源，发现 KafkaSpout 代码里（storm.kafka.KafkaSpout）使用了 slf4j 的包，而 Kafka 系统本身（kafka.consumer.SimpleConsumer）却使用了 apache 的包，这个结果着实有些尴尬。

折腾了一会儿，最后还是根据 http://stackoverflow.com/questions/20117720/detected-both-log4j-over-slf4j-jar-and-slf4j-log4j12-jar-on-the-class-path-pree 这个问题的提示，在依赖定义中排除问题依赖包（也就是 Kafka 本身的依赖包）中对应的冲突的包

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka_2.10</artifactId>
    <version>0.8.2.1</version>
    <scope>provided</scope>
    <exclusions>
        <exclusion>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-log4j12</artifactId>
        </exclusion>
        <exclusion>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
        </exclusion>
    </exclusions>
</dependency>

重新编译运行就 OK 了。

TIPS

结合这次经历和以前遇到的各种 log4j 问题（没办法，使用最广泛的包就是这么任性），总结了一下使用 log4j 的几点不算什么经验的心得。

统一采用 slf4j 的包

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class MyBolt {
    private static final Logger LOG = LoggerFactory
            .getLogger(MyBolt.class);
}

在代码中不再使用 apache 的包，转而统一使用 slf4j 的包，可以避免大多数包冲突问题（只适用于 Storm 相关服务，这是因为 Storm 所依赖的 logback-classic-1.0.13.jar 是使用 slf4j 的）。

Maven 的编译发布插件中最好都要排除冲突的包

<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.1</version>
        <configuration>
            <source>1.7</source>
            <target>1.7</target>
            <excludes>
                <exclude>log4j:log4j:jar:</exclude>
            </excludes>
        </configuration>
    </plugin>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <executions>
            <execution>
                <phase>package</phase>
                <goals>
                    <goal>shade</goal>
                </goals>
            </execution>
        </executions>
        <configuration>
            <finalName>${project.artifactId}-${project.version}-shade</finalName>
            <artifactSet>
                <excludes>
                    <exclude>log4j:log4j:jar:</exclude>
                </excludes>
            </artifactSet>
            <transformers>
                <transformer
                    implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
                <transformer
                    implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                    <mainClass>storm.test.topology.SimulationTopology</mainClass>
                </transformer>
            </transformers>
        </configuration>
    </plugin>
</plugins>

<exclude>log4j:log4j:jar:</exclude> 这样的操作可以避免拓扑发布到 Storm 集群之后可能出现的包冲突问题。

TIPS

Comments