Kafka Producer Configuration

Introduction

From maximising throughput, to ensuring idempotency, to achieving exactly-once messaging, there are many factors to consider when configuring an application’s Kafka Producer. The following are some of the most important to consider. Drill into the corresponding articles for the full details on each.

Idempotent Producer

Transient errors such as network connection issues can mean that the Kafka broker fails to acknowledge whether a message has been successfully written, leaving the Producer unsure if that message has been emitted. To ensure no message loss occurs if the write did fail the Producer must retry. By configuring the Producer to be idempotent, it is guaranteed that the message will only be written once, with no duplicate writes occurring.

Read more: Idempotent Producer

Producer Acknowledgements

A Producer is able to consider a message successfully written to a topic partition when it has received the expected acknowledgements from the Kafka broker. The number of acknowledgements required from the lead and replica partitions is determined by the Producer acks configuration. The fewer acknowledgements required the greater the availability and throughput, but this comes at the risk of durability with possible message loss in failure scenarios.

Read more: Producer Acks

Producer Message Batching

Message throughput can be maximised by optimising the message batch size that the Producer uses when sending requests to the broker. This comes at the expense of latency which is exacerbated if batches are not filled within the configured period of time, so careful measuring and tuning is required.

Read more: Producer Message Batching

Transactional Producer

When a Producer emits events using Kafka Transactions, the write to the topic partition can be guaranteed to be committed, or aborted, only once. This will happen atomically with the update to the consumer offsets topic. Downstream transaction aware consumers are then guaranteed to receive a committed message exactly-once.

Read more: Kafka Transactions

Message Ordering

In order to maximise throughput a Producer can be configured to have multiple unacknowledged requests in flight simultaneously. When Producer retries are enabled for transient errors there is the possibility that ordering can be lost. For a non-idempotent Producer this means that message ordering is not guaranteed. For an idempotent Producer ordering is guaranteed for up to a maximum of five inflight requests.

Read more: Idempotent Producer - Message Ordering

Introduction to Kafka with Spring Boot   

Lydtech's Udemy course Introduction to Kafka with Spring Boot covers everything from the core concepts of messaging and Kafka through to step by step code walkthroughs to build a fully functional Spring Boot application that integrates with Kafka. Put together by our team of Kafka and Spring experts, this course is the perfect introduction to using Kafka with Spring Boot.

View this article on our Medium Publication.