Getting started with Apache Kafka in less than 30 minutes

Apache Kafka Getting Started Tutorial

Apache Kafka is Open source most used messaging system in Big Data environment. This software was developed at LinkedIn and later on released as Open source project at Apache Foundation.

Apache Kafka latest release at the time of writing of this tutorial was 1.0.0 and it is distributed on the Apache Foundation Kafka website. You can download binary version for this tutorial and get started quickly.

Apache Kafka is open source publish subscribe messaging system developed in Java and Scala.

I am using Ubuntu 16.04 for installing Kafka and running examples. I am assuming that you have access to Ubuntu desktop and JDK 8 or above is installed.

Let's get started with Apache Kafka.

Step 1: Login to Ubuntu Desktop and check Java version

java -version

It should print following message:

eepak@deepak-VirtualBox:~$ java -version
openjdk version "1.8.0_91"
OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-3ubuntu1~16.04.1-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
deepak@deepak-VirtualBox:~$

If you don't have Java installed on your system then install it. You can check tutorial: How to Install Oracle Java JDK 8 in Ubuntu 16.04?

Kafka will work with both OpenJDK and Oracle JDK. So, you ca use any of the JDK.

Step 2: Download and install Kafka

For this tutorial we will download Kafka using wget on Ubuntu, extract and then run in terminal. Make a new directory kafka and go that directory and download Kafka. Here is command:

mkdir kafka
cd kafka
wget http://www-eu.apache.org/dist/kafka/1.0.0/kafka_2.11-1.0.0.tgz

After downloading unzip it with following command:

tar -xzvf kafka_2.11-1.0.0.tgz

Now change to the kafka directory with the command cd kafka_2.11-1.0.0/

Step 3: Start ZooKeeper and Kafka Server

Apache Kafka uses ZooKeeper and it should be running before running Kafka server. Run following command in Ubuntu terminal.

Make sure you are in kafka_2.11-1.0.0/ directory.

Command to start ZooKeeper server:

bin/zookeeper-server-start.sh config/zookeeper.properties

After starting ZooKeeper server use following command to start Kafka in another terminal.

bin/kafka-server-start.sh config/server.properties

Step 4: Create Topic

Now we will create create topic with following command:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Above command will create a topic with the name "test". Now you can get list of all topics with following command:

bin/kafka-topics.sh --list --zookeeper localhost:2181

Above command list of all the topics on server, in our case its "test".

Step 5: Sending message to test Topic

Now we will send message to the test topic. Here is the command:

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

Above command will provide you a prompt and you can write messages.

Here is sample:

deepak@deepak-VirtualBox:~/kafka/kafka_2.11-1.0.0$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>RoseIndia.net Kafka Tutorial
>Learn Kafka 
>Welcome to Kafka server
>^Cdeepak@deepak-VirtualBox:~/kafka/kafka_2.11-1.0.0$

Step 6: Receiving messages

Now we will get all the messages sent to test topic with following command:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

Here is sample output:

deepak@deepak-VirtualBox:~/kafka/kafka_2.11-1.0.0$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
RoseIndia.net Kafka Tutorial
Learn Kafka 
Welcome to Kafka server
^CProcessed a total of 3 messages
deepak@deepak-VirtualBox:~/kafka/kafka_2.11-1.0.0$

In this you learned how to install Kafka server and user it for playing with messaging. Check more tutorials at Apache Kafka Tutorials.