Let's get started with the Apache Kafka in in less then 30 minutes and learn to install, create topic, send message and receive message.Let's get started with the Apache Kafka in in less then 30 minutes and learn to install, create topic, send message and receive message.
Apache Kafka is Open source most used messaging system in Big Data environment. This software was developed at LinkedIn and later on released as Open source project at Apache Foundation.
Apache Kafka latest release at the time of writing of this tutorial was 1.0.0 and it is distributed on the Apache Foundation Kafka website. You can download binary version for this tutorial and get started quickly.
Apache Kafka is open source publish subscribe messaging system developed in Java and Scala.
I am using Ubuntu 16.04 for installing Kafka and running examples. I am assuming that you have access to Ubuntu desktop and JDK 8 or above is installed.
Let's get started with Apache Kafka.
Login to your your Ubuntu desktop and check if java is installed with following command:
It should print following message:
[email protected]:~$ java -version openjdk version "1.8.0_91" OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-3ubuntu1~16.04.1-b14) OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) [email protected]:~$
If you don't have Java installed on your system then install it. You can check tutorial: How to Install Oracle Java JDK 8 in Ubuntu 16.04?
Kafka will work with both OpenJDK and Oracle JDK. So, you ca use any of the JDK.
For this tutorial we will download Kafka using wget on Ubuntu, extract and then run in terminal. Make a new directory kafka and go that directory and download Kafka. Here is command:
mkdir kafka cd kafka wget http://www-eu.apache.org/dist/kafka/1.0.0/kafka_2.11-1.0.0.tgz
After downloading unzip it with following command:
tar -xzvf kafka_2.11-1.0.0.tgz
Now change to the kafka directory with the command cd kafka_2.11-1.0.0/
Apache Kafka uses ZooKeeper and it should be running before running Kafka server. Run following command in Ubuntu terminal.
Make sure you are in kafka_2.11-1.0.0/ directory.
Command to start ZooKeeper server:
After starting ZooKeeper server use following command to start Kafka in another terminal.
Now we will create create topic with following command:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Above command will create a topic with the name "test". Now you can get list of all topics with following command:
bin/kafka-topics.sh --list --zookeeper localhost:2181
Above command list of all the topics on server, in our case its "test".
Now we will send message to the test topic. Here is the command:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
Above command will provide you a prompt and you can write messages.
Here is sample:
[email protected]:~/kafka/kafka_2.11-1.0.0$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test >RoseIndia.net Kafka Tutorial >Learn Kafka >Welcome to Kafka server >^[email protected]:~/kafka/kafka_2.11-1.0.0$
Now we will get all the messages sent to test topic with following command:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
Here is sample output:
[email protected]:~/kafka/kafka_2.11-1.0.0$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning RoseIndia.net Kafka Tutorial Learn Kafka Welcome to Kafka server ^CProcessed a total of 3 messages [email protected]:~/kafka/kafka_2.11-1.0.0$
In this you learned how to install Kafka server and user it for playing with messaging. Check more tutorials at Apache Kafka Tutorials.