Kafka Topics: Fat Pipe Or Thin Pipe?

Introduction

Whether you're experienced with Kafka or setting out on your first adventure, one question is almost certain to be raised, "Should a topic be used for more than one event type?"

As with most architectural decisions, the answer is of course 'it depends'. To help understand what is best fit and matches your requirements we'll first take a look at the two extremes to understand what we may want to avoid.

The Single Topic

One topic to rule them all. In this approach all events share a topic. It's most likely that this is not the solution you are looking for, but why not?

Focus

Each consumer is most likely a service, and a good service design is that the behaviour of a service is focused, the service should do one thing and do it well. With the single pipe approach each service is consuming all events. Having access to these events is a source of temptation which could see your services easily losing their focus.

Resource Wastage

Consuming all events also has the downside of having the service active and processing more often than is necessary. Even if that processing is just discarding an event, it's processing that does not need to be done. This unnecessary work is not restricted to the consuming service, network bandwidth will also be consumed by all these unwanted messages.

Event/Schema Validation

With so many event types on the same topic the challenges of validating the event against a given schema become trickier. Each event can contain its own schema (a common pattern) or alternatively each message could contain a reference to the schema in the schema registry. Even so, having many different schemas per topic should make you rethink the purpose of that topic.

System Understandability

Understanding of the system. By throwing all your events down a single pipe, you lose visibility of the events occurring in the system. Decommissioning production of events, it becomes difficult to ascertain which services are consuming the events.

Discrete Topics

This approach sees each event type having its own topic. On the surface this doesn't feel like a bad choice, as a consumer you get complete control over which events you are looking for. Are there any gotchas with this approach?

Order of Events

Ordering necessitates that events share the same partition, to share a partition events must use the same key. Ordering of events across different partitions isn't guaranteed. When using different topics (or partitions within a topic) circumstances can conspire against you, and you may find that events arrive in an illogical order. There are inventive workarounds, but this is extra work you shouldn't be doing if you can avoid it. However, if ordering isn't important, then that argument does not apply.

Performance at Scale

There is no defined limit to the number of topics you may use, however, the number of topics and the number of partitions are linked. As a guideline the recommendation is not to exceed 4000 partitions per broker. If you're expecting to get anywhere near these figures, then reducing the number of topics will be a consideration.

Middle Ground?

One advantage of examining the extremes first is that it helps us clearly identify the behaviours that really matter.. The severity of these issues highlighted in the extremes will change depending on the implementation, even to an extent where a perceived downside may be an acceptable risk.

From what we've reviewed so far there are a number of qualities we require of our system;

Focus, Understandability, Testability and Maintainability. We probably also want to add efficiency to that list if that's achievable. Already, this is suggesting that we should be using more topics.

There are two characteristics that will guide our decision more than most; Ordering, and Security.

Ordering is an important characteristic we need to address. It makes sense that changes to an entity must occur in a given order, for example, if an order is to be cancelled, then that order must first have been created. Conversely, there's often little value in ordering the events on unrelated entities. For example the cancellation of an order does not necessarily have a bearing on the updating of an email address, therefore email address updates and order updates belong in different topics.

Data security is another consideration that may influence your decision to share topics. Does the event contain sensitive information which should not be exposed to all but a few services? A payment for example, the new payment may be completed, but there is no need to share all the payment details. Maybe you need to send only a subset of the information, such as the new state and the entity id. This requirement should be satisfied by service design.

Summing up

There is no one answer and the direction you choose will depend on the size/scale of your development, as well as important calls on ordering and visibility of data, but it is reasonable to assume your solution will comprise a number of shared and discrete topics.

---

View this article on our Medium Publication.