How Fast Can We Insert? A Performance Study of Apache Kafka

Message brokers see widespread adoption in modern IT landscapes. These systems are comparatively easy to use and configure and thus, present a flexible solution for various data storage scenarios. Their ability to scale horizontally enables users to adapt to growing data volumes and changing environments. However, one of the main challenges concerning message brokers is the danger of them becoming a bottleneck within an IT architecture. To prevent this, the amount of data a given message broker can handle with a specific configuration needs to be known. In this paper, we propose a monitoring architecture for message brokers and similar systems. We present a comprehensive performance analysis of Apache Kafka, a popular message broker implementation, using our approach. As part of the benchmark, we study selected data producer settings and their impact on the achievable data ingestion rate.