Lydtech Consulting - Kafka on Kubernetes - Part 3: Spring Boot Demo

Introduction

This is the final part in the three part series exploring deploying and integrating with Kafka on Kubernetes. In the first part an overview of Kubernetes, minikube and kubectl was provided, and in the second part the steps to deploy Kafka and Zookeeper were described. In this part the steps to add a Spring Boot application to the deployment are walked through.

The application has a REST endpoint that can be called to trigger sending events to Kafka. The steps to create an ingress for the application are covered. This allows a client to call the application via a standard URL rather than having to use the IP and port combination. Events can also be sent to Kafka that the application consumes. These trigger outbound events to be produced by the application, and the steps to achieve this and consume these outbound events are described.

The source code for the accompanying Spring Boot application is available here.

Kubernetes Deployment

A Spring Boot application will be deployed alongside Kafka and Zookeeper in the Kubernetes cluster. The Spring Boot application is deployed in a Docker container, and a Kubernetes pod provides an abstraction creating a running environment for the container. A Kubernetes service is attached to the pod, and this provides a static IP address and performs load balancing. This service then communicates with Kafka via that application’s service. In order to make the Spring Boot service accessible to an external source via a standard URL, an ingress is created.

Figure 1: Kubernetes deployment

Spring Boot Application Overview

The Spring Boot application that will be deployed into the Kubernetes cluster offers a REST endpoint enabling a client to request that the specified number of events are emitted to an outbound Kafka topic. This endpoint will be used in the demo to prove that an external client can call the deployed application successfully. It also exposes an endpoint to return the application version, which is useful when verifying that the application can be hit.

Figure 2: The Spring Boot demo application

The application also has a Kafka consumer listening on an inbound topic. The events on this topic likewise trigger the application into emitting the requested number of events. The demo will show sending events to the Kafka instance deployed in Kubernetes, and then consuming the outbound events emitted by the application using the Kafka command line tools.

Create Kafka Topics

The Spring Boot demo application uses two topics, demo-inbound-topic and demo-outbound-topic. These should be created in advance. Remember that all kubectl commands should be run in the demo namespace. This can be configured as the default namespace for the commands, as covered in Part 2.

First jump onto the Kafka pod:

kubectl exec -it kafka-deployment-7985656cd5-8f8r7 -- /bin/bash

Then execute the following commands to create the topics:

kafka-topics --bootstrap-server localhost:9092 --create --topic demo-inbound-topic --replication-factor 1 --partitions 3
kafka-topics --bootstrap-server localhost:9092 --create --topic demo-outbound-topic --replication-factor 1 --partitions 3

Build and Deploy the Spring Boot Application

The bootstrap-servers URL in the Spring Boot application properties (src/main/resources/application.yml) is configured to point to the deployed Kafka service:

kafka:
    bootstrap-servers: kafka-service:9092

The Kubernetes deployment manifest for the Spring Boot application is located in resources/demo-application.yml.

The Spring Boot application is built using maven:

mvn clean install

Kubernetes uses its own local Docker registry that is not connected to the Docker registry on the local machine. As the Spring Boot application Docker image will be built locally and is not available in the public Docker registry, the imagePullPolicy is set to Never in the demo-application.yml. The following minikube command is then required in order to output the environment variables that are needed to point the local Docker daemon to the minikube internal Docker registry:

eval $(minikube -p minikube docker-env)

This command must be run in any new terminal window opened to take effect for the next Docker command.

Then the Docker image can be built for the application. This will be installed into the minikube registry, ready for deployment to Kubernetes:

docker build -t kube/kafka-kubernetes-demo:latest .

Finally the Spring Boot pod can be created and deployed:

kubectl create -f ./resources/demo-application.yml

Console output shows the deployment and service are created:

deployment.apps/kafka-kubernetes-demo created
service/kafka-kubernetes-demo-service created

kafka-kubernetes-demo-service is the name defined in the service metadata name from the demo-application.yml:

metadata:
  name: kafka-kubernetes-demo-service

The job can be removed and recreated if necessary, by first deleting it:

kubectl delete -f ./resources/demo-application.yml

View the dashboard to see the details on the deployment, ensuring the demo namespace is selected. Alongside the existing Kafka and Zookeeper deployments the new Spring Boot application dashboard can be observed:

Figure 3: minikube dashboard

Expose the Spring Boot Application

In order to call the application from an external source, assign an external port by starting a tunnel (the namespace must be included in this command):

minikube service kafka-kubernetes-demo-service --namespace demo

Figure 4: minikube service tunnel

This will open the root page of the Spring Boot application in the browser, which displays Spring Boot Demo, verifying the application has started successfully.

Figure 5: Spring Boot application landing page

This is the URL to use to call the service from an external source. Alternatively to get the URL without opening the browser, use the --url option:

minikube service kafka-kubernetes-demo-service --namespace demo --url

Access Application via an Ingress

In order to expose the Spring Boot application via a standard URL rather than just the IP and port as with the service, an ingress can be created. To support an ingress, minikube needs this addon enabled:

minikube addons enable ingress

It can be verified by running:

kubectl get pods -n ingress-nginx

Figure 6: Ingress nginx pods

Next apply an ingress for the kafka-kubernetes-demo-service, which is the name of the Spring Boot application service derived from the demo-application.yml. The ingress for the Spring Boot application is provided in the root of the project under the resources directory: demo-ingress.yml.

kubectl apply -f ./resources/demo-ingress.yml

The output should show this has created successfully:

ingress.networking.k8s.io/demo-ingress created

To view the ingress use the command:

kubectl get ingress

The output shows the host name as declared in the demo-ingress.yml file, k8s.springboot.demo:

Figure 7: Spring Boot application ingress

If the ingress should be deleted, the command to run is:

kubectl delete ingress demo-ingress

In order to send requests to this host, the /etc/hosts file should be updated with the following line (correct for MacOS ARM) which will then forward requests to this URL to minikube:

127.0.0.1 k8s.springboot.demo

For other operating systems add the minikube IP to the /etc/hosts file instead, which is the IP from the get ingress command above:

192.168.49.2 k8s.springboot.demo

Now start the minikube tunnel:

minikube tunnel

In the browser navigate to the following URL to see the landing page:

http://k8s.springboot.demo

Figure 8: Spring Boot application landing page

Running the Demo

Interacting via REST

With an ingress applied and the minikube tunnel open as described above, hit the version endpoint on the Spring Boot application using curl:

curl -X GET http://k8s.springboot.demo/v1/demo/version

Alternatively if the ingress has not been created then the service can be hit using the IP and port (with the service tunnel open):

minikube service kafka-kubernetes-demo-service --namespace demo

curl -X GET http://localhost:60921/v1/demo/version

Either should return v1.

In order to trigger the application into sending three outbound events to Kafka, send a REST request to the following URL:

curl -v -d '{"numberOfEvents":3}' -H "Content-Type: application/json" http://k8s.springboot.demo/v1/demo/trigger

The response should be a 200 Success. Check the Spring Boot application logs to confirm events have been sent, acquiring the pod name from the get pods command:

kubectl logs kafka-kubernetes-demo-745d47966c-f68hj

13:23:34.320 INFO  d.k.KafkaDemoApplication - Started KafkaDemoApplication in 1.423 seconds (process running for 1.648)
14:14:43.734 INFO  d.k.s.DemoService - Sending 3 events
14:14:45.318 INFO  d.k.s.DemoService - Total events sent: 3

Interacting via Events

Jump onto the Kafka pod:

kubectl exec -it kafka-deployment-7985656cd5-8f8r7 -- /bin/bash

Consume the events already produced from the beginning of the demo-outbound-topic:

kafka-console-consumer --bootstrap-server localhost:9092 --topic demo-outbound-topic --from-beginning

The events that were emitted from the call to the REST endpoint, populated with randomised names, are output in the console:

{"firstName":"ZrdOwISesV","middleName":"FXjMEVAyZf","lastName":"WRfyXxHSGB"}
{"firstName":"qTZORIeXpU","middleName":"qCurtqohni","lastName":"mWUIEeTSQk"}
{"firstName":"xvlSzBpKVZ","middleName":"HjNQjirBcu","lastName":"MwzFClWjHV"}

Trigger sending new events from the application by submitting a request to the demo-inbound-topic, and observe the new events being emitted. Use a new terminal window in order to leave the kafka-console-consumer running.

kafka-console-producer --broker-list localhost:9092 --topic demo-inbound-topic 
{"numberOfEvents":2}

The new outbound events emitted should be consumed by the kafka-console-consumer. This send to Kafka should also be reflected in the Spring Boot application logs:

14:25:09.789 INFO  d.k.c.KafkaDemoConsumer - Received message - event: DemoInboundEvent(numberOfEvents=2)
14:25:09.789 INFO  d.k.s.DemoService - Sending 2 events
14:25:09.818 INFO  d.k.s.DemoService - Total events sent: 2

Summary

This series of articles started by describing the key concepts of the container orchestration tool Kubernetes, and then applied this knowledge by using minikube and kubectl to run and interact with a local Kubernetes cluster. It then stepped through deploying Kafka and Zookeeper, before deploying, exposing and interacting with a Spring Boot application in this final part.

Source Code

The source code for the accompanying Spring Boot demo application is available here:

https://github.com/lydtechconsulting/kafka-kubernetes-demo/tree/v1.0.0

Kafka on Kubernetes - Part 3: Spring Boot Demo