Chapter 1: Introduction to Apache Kafka

In this section, we will explore how to integrate Apache Kafka into our Spring Boot application. Apache Kafka serves as a distributed streaming platform that excels in real-time data handling and supports event-driven architectures. When combined with Spring Boot, a robust framework for Java applications, it provides an efficient solution.

Chapter 2: Setting Up a Test Kafka Cluster

To get started with Kafka, you can quickly set up a local Kafka cluster on your PC. Docker is also a viable option for this setup. Be aware that older Kafka versions require Apache Zookeeper to function.

In this guide, we will use Docker Compose to initiate Kafka, as outlined in Mahdi Mallaki's article. This setup also provides a user-friendly interface.

To launch the Docker Compose setup, run the following command:

docker-compose -f docker-compose.yaml up -d

To shut down the Kafka cluster, use:

docker-compose -f kafka-docker-compose.yaml down

Ensure you have Docker and Docker Compose installed. After launching, verify the setup by visiting http://localhost:8080 in your browser to access the Kafka UI.

Chapter 3: Creating a Topic in Kafka

Within the Kafka UI, select the 'Topics' tab and click 'Add a Topic'. Enter the required parameters and click 'Create'. For instance, you might create a test topic with three partitions, ensuring redundancy by having two brokers store each partition.

Chapter 4: Integrating with the Spring Kafka Module

Spring Boot facilitates integration with Apache Kafka through its Spring Kafka module, providing a user-friendly abstraction over the Apache Kafka Java client library. This simplifies the configuration and management of messages within a Spring Boot application.

Key components of Spring Kafka include:

• KafkaTemplate: A high-level abstraction for sending messages to Kafka topics, offering both synchronous and asynchronous methods.
• @KafkaListener: An annotation for defining message listener methods that automatically subscribe to Kafka topics.
• ProducerFactory and ConsumerFactory: Interfaces for creating instances of Kafka producers and consumers, respectively.
• ConcurrentMessageListenerContainer: A container that allows for concurrent message consumption from Kafka topics.

Chapter 5: Writing the Code

To begin, create two separate projects through Spring Initializr: one for the producer and another for the consumer. Ensure you include Spring for Apache Kafka and Spring Web as dependencies.

In the producer/src/main/resources/application.properties file, add:

spring.kafka.bootstrap-servers=localhost:9192

This configuration points Spring to the Kafka broker's URL, allowing it to identify other brokers automatically.

Producer Application:

Create a file named KafkaProducer.java and implement the following code:

import org.springframework.beans.factory.annotation.Autowired;

import org.springframework.kafka.core.KafkaTemplate;

import org.springframework.stereotype.Component;

@Component

public class KafkaProducer {

@Autowired

private KafkaTemplate<String, String> kafkaTemplate;

public void sendMessage(String topic, String key, String message) {

kafkaTemplate.send(topic, key, message);

System.out.println("Message sent: " + message);

}

}

Explanation of the Producer Code:

• KafkaTemplate: This class is essential for sending messages to Kafka topics, parameterized with types for keys and values.
• sendMessage Method: This method sends messages to the specified Kafka topic, taking in the topic name, an optional key, and the message itself.

Message Controller:

Now, create a MessageController.java file with the following content:

import org.springframework.beans.factory.annotation.Autowired;

import org.springframework.web.bind.annotation.GetMapping;

import org.springframework.web.bind.annotation.RequestParam;

import org.springframework.web.bind.annotation.RestController;

@RestController

public class MessageController {

private final KafkaProducer kafkaProducer;

@Autowired

public MessageController(KafkaProducer kafkaProducer) {

this.kafkaProducer = kafkaProducer;

}

@GetMapping("/send")

public String sendMessageToKafka(@RequestParam String message) {

kafkaProducer.sendMessage("test", "test", message);

return "Message sent to Kafka: " + message;

}

}

This code defines a REST controller that processes GET requests at the /send endpoint, sending messages to a Kafka topic using the KafkaProducer instance.

To run the producer, execute:

mvn spring-boot:run

Then visit http://localhost:8081/send?message=Hello to see your message appear in the Kafka UI.

Chapter 6: Final Thoughts

This guide provided an introduction to integrating Apache Kafka with Spring Boot, covering essential concepts such as the publish-subscribe model, distributed architecture, fault tolerance, and scalability. Utilizing Kafka alongside Spring Boot offers a robust framework for developing real-time data processing and event-driven applications.

For further insights into Kafka's capabilities, explore more configurations within the Spring Kafka module.

This video provides a comprehensive crash course on Apache Kafka integration with Spring Boot 3.0.x, helping you get started with practical examples.

In this video, learn how to effectively use Spring for Apache Kafka, featuring insights from expert Viktor Gamov to enhance your understanding.