Getting started with Apache Kafka in less than 30 minutes

Ads
 

Getting started with Apache Kafka in less than 30 minutes

Let's get started with the Apache Kafka in in less then 30 minutes and learn to install, create topic, send message and receive message.

Let's get started with the Apache Kafka in in less then 30 minutes and learn to install, create topic, send message and receive message.

Apache Kafka Getting Started Tutorial

Apache Kafka is Open source most used messaging system in Big Data environment. This software was developed at LinkedIn and later on released as Open source project at Apache Foundation.

Apache Kafka latest release at the time of writing of this tutorial was 1.0.0 and it is distributed on the Apache Foundation Kafka website. You can download binary version for this tutorial and get started quickly.

Apache Kafka is open source publish subscribe messaging system developed in Java and Scala.

I am using Ubuntu 16.04 for installing Kafka and running examples. I am assuming that you have access to Ubuntu desktop and JDK 8 or above is installed.

Let's get started with Apache Kafka.

Step 1: Login to Ubuntu Desktop and check Java version

Login to your your Ubuntu desktop and check if java is installed with following command:

java -version

It should print following message:

[email protected]:~$ java -version
openjdk version "1.8.0_91"
OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-3ubuntu1~16.04.1-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
[email protected]:~$ 

If you don't have Java installed on your system then install it. You can check tutorial: How to Install Oracle Java JDK 8 in Ubuntu 16.04?

Kafka will work with both OpenJDK and Oracle JDK. So, you ca use any of the JDK.

Step 2: Download and install Kafka

For this tutorial we will download Kafka using wget on Ubuntu, extract and then run in terminal. Make a new directory kafka and go that directory and download Kafka. Here is command:

mkdir kafka
cd kafka
wget http://www-eu.apache.org/dist/kafka/1.0.0/kafka_2.11-1.0.0.tgz

After downloading unzip it with following command:

tar -xzvf kafka_2.11-1.0.0.tgz

Now change to the kafka directory with the command cd kafka_2.11-1.0.0/

Step 3: Start ZooKeeper and Kafka Server

Apache Kafka uses ZooKeeper and it should be running before running Kafka server. Run following command in Ubuntu terminal.

Make sure you are in kafka_2.11-1.0.0/ directory.

Command to start ZooKeeper server:

bin/zookeeper-server-start.sh config/zookeeper.properties

After starting ZooKeeper server use following command to start Kafka in another terminal.

bin/kafka-server-start.sh config/server.properties

Step 4: Create Topic

Now we will create create topic with following command:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Above command will create a topic with the name "test". Now you can get list of all topics with following command:

bin/kafka-topics.sh --list --zookeeper localhost:2181

Above command list of all the topics on server, in our case its "test".

Step 5: Sending message to test Topic

Now we will send message to the test topic. Here is the command:

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

Above command will provide you a prompt and you can write messages.

Here is sample:

[email protected]:~/kafka/kafka_2.11-1.0.0$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>RoseIndia.net Kafka Tutorial
>Learn Kafka 
>Welcome to Kafka server
>^[email protected]:~/kafka/kafka_2.11-1.0.0$ 

Step 6: Receiving messages

Now we will get all the messages sent to test topic with following command:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

Here is sample output:

[email protected]:~/kafka/kafka_2.11-1.0.0$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
RoseIndia.net Kafka Tutorial
Learn Kafka 
Welcome to Kafka server
^CProcessed a total of 3 messages
[email protected]:~/kafka/kafka_2.11-1.0.0$ 

In this you learned how to install Kafka server and user it for playing with messaging. Check more tutorials at Apache Kafka Tutorials.

Ads

Ads