我要啦免费统计

Set Up Kafka On Windows (Actually Working)

In this tutorial, I will show how to set up and run Apache Kafka on Windows. Getting Kafka 0.8.1 running on Windows isn’t straightforward if you follow Apache Kafka’s documentation. They are somewhat misleading, and the bat files included in the Kafka package are significantly outdated and could not work properly. However, with correct instructions and updated bat files it’s easy and can be done under 10 minutes.

Kafka comes with two sets of scripts to run Kafka. In the bin folder, the sh files are used to set up Kafka in a Linux environment. In the bin\windows folder, there are also some bat files corresponds to those sh files which are supposed to work in a Windows environment. But these bat files are not well-maintained. For example, Kafka sh scripts merged kafka-list-topic.sh and kafka-create-topic.sh to kafka-topics.sh instead ago; but the windows bat files still uses the old kafka-list-topics.bat and kafka-create-topic.sh. Some say you can use Cygwin to execute the sh scripts in order to run Kafka. However, they are many additional steps involved, and in the end you may not get the desired outcome. With the correct bat files, there is no need to use Cygwin, and only Server JRE is required to run Kafka on Windows.

Step 0: Preparation

Install Java SE Server JRE

You need Java SE Server JRE in order to run Kafka. If you have JDK installed, you already have Server JRE installed, just check if the folder {JRE_PATH}\bin\server exists. If it is not, follow the following steps to install Java SE Server JRE:

  1. Download Java SE Server JRE from http://www.oracle.com/technetwork/java/javase/downloads/index.html

  2. Unpack it to a folder, for example C:\JREServer.

  3. Update the system environment variable PATH to include C:\JREServer\jre\bin, follow this guide provided by Java.

Download Kafka

  1. Download the binaries from http://kafka.apache.org/downloads.html

  2. Unpack it to a folder, for example C:\kafka

Step 1: Update files and configurations

Update the bat files to correct version

Unfortunately the bat files for Kafka 0.8.1 are full of errors and are significantly outdated, so in order to make Kafka running correctly, they must be replaced.

  1. Download updated Windows bat files from https://github.com/Anilot/kafka/releases/

  2. Unzip and copy them into C:\kafka\bin\windows (overwrite the files already there)

Update Kafka configuration files

The config files need to be updated corresponding to Windows path naming convention.

  1. Open config\server.properties, change
server.properties
1
log.dirs=/tmp/kafka-logs

to

server.properties
1
log.dirs=c:/kafka/kafka-logs

2.. Open config\zookeeper.properties, change

zookeeper.proerties
1
dataDir=/tmp/zookeeper

to

zookeeper.properties
1
dataDir=c:/kafka/zookeeper-data

Step 2: Start the Server

In Windows Command Prompt, switch the current working directory to C:\kafka:

1
cd C:\kafka
  1. Start Zookeeper

Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you don’t already have one. You can use the convenience script packaged with Kafka to get a quick-and-dirty single-node ZooKeeper instance.

1
2
3
> .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
[2014-08-05 15:01:37,495] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
...

2.. Start Kafka in another Command Prompt or Powershell Console

1
> .\bin\windows\kafka-server-start.bat .\config\server.properties

Step 3: Create a topic

  1. Create a topic

Let’s create a topic named “test” with a single partition and only one replica:

1
> .\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

2.. List topics

We can now see that topic if we run the list topic command:

1
> .\bin\windows\kafka-topics.bat --list --zookeeper localhost:2181

Step 4: Send some messages

Kafka comes with a command line client that will take input from a file or from standard input and send it out as messages to the Kafka cluster. By default each line will be sent as a separate message.

  1. Start console producer
1
> .\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test

2.. Write some messages

1
2
This is a message
This is another message

Step 5: Start a consumer

1
> .\bin\windows\kafka-console-consumer.bat --zookeeper localhost:2181 --topic test --from-beginning

If you have each of the above commands running in a different terminal then you should now be able to type messages into the producer terminal and see them appear in the consumer terminal.

Yay cheers!


Common Errors and Solutions

Could not find or load main class org.apache.zookeeper.server.quorum.QuorumPeerMain

If you have encountered the following error:

1
Error: Could not find or load main class org.apache.zookeeper.server.quorum.QuorumPeerMain

It means you are using the incorrect Windows bat files. Follow the steps above to get the updated version of Windows Bat files.

Error when freeing index buffer (kafka.log.OffsetIndex)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[2014-08-06 10:50:35,005] WARN Error when freeing index buffer (kafka.log.OffsetIndex)
java.lang.NullPointerException
at kafka.log.OffsetIndex.kafka$log$OffsetIndex$$forceUnmap(OffsetIndex.scala:301)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:283)
at kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
at kafka.utils.Utils$.inLock(Utils.scala:538)
at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
at kafka.log.Log.loadSegments(Log.scala:157)
at kafka.log.Log.<init>(Log.scala:64)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:118)
at kafka.log.LogManager$$anonfun$loadLogs$1$$anonfun$apply$4.apply(LogManager.scala:113)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:38)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:113)
at kafka.log.LogManager$$anonfun$loadLogs$1.apply(LogManager.scala:105)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:33)
at kafka.log.LogManager.loadLogs(LogManager.scala:105)
at kafka.log.LogManager.<init>(LogManager.scala:57)
at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:275)
at kafka.server.KafkaServer.startup(KafkaServer.scala:72)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)

You need to update the Windows bat files.

Failed to load class “org.slf4j.impl.StaticLoggerBinder”

kafka-console-producer.sh (kafka_2.09-0.8.1) reported SLF4J: Failed to load class “org.slf4j.impl.StaticLoggerBinder” error as below:

1
2
3
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

The reason is that in the kafka libs directory, the kafka release zip file only included jar file of slf4j-api, they missed a jar file: slf4j-nop.jar, so we have to go to http://www.slf4j.org/download.html and download slf4j-1.7.7.zip, and then unzip it, copy slf4j-api-1.7.7.jar and slf4j-nop-1.7.7.jar into kafka’s libs directory. restart kafka producer again, now no error is reported. (Credit)

References

If you have any other questions, feel free to leave a comment below.