MQTT vs Kafka: Device Messaging vs Event Streaming

MQTT vs Kafka is one of the most common architecture questions in IoT, and the honest answer is that they solve different halves of the problem. MQTT gets data off thousands of small, intermittently connected devices; Kafka stores, replays and processes high-volume event streams inside your data platform. This guide compares the two, explains where each one breaks down, and shows a working pattern for moving data from an MQTT broker into Kafka with a consumer service you control.

Two Tools for Two Jobs

MQTT is a lightweight publish/subscribe protocol. Devices open one TCP connection to a broker, publish to hierarchical topics and subscribe to the topics they need. The broker routes messages in real time, keeps the last retained value per topic, buffers QoS 1 and 2 messages for disconnected persistent sessions and publishes a Last Will when a device drops. It is designed for constrained clients and unreliable networks. If you are new to it, what an MQTT broker does is a good starting point.

Apache Kafka is a distributed, append-only commit log. Producers write records to partitioned topics, Kafka stores them durably for a configurable retention period, and consumer groups read at their own pace and can rewind to reprocess history. It is designed for backend services on reliable networks moving large volumes of events between systems, with ecosystem tools for stream processing and connectors to databases and data lakes.

The confusion comes from the shared vocabulary. Both have "topics", both have producers and consumers, and both are called messaging systems. But an MQTT topic is a routing address that can be created on the fly with thousands of levels of variation, while a Kafka topic is a durable, partitioned log that you provision deliberately.

MQTT vs Kafka at a Glance

Why Devices Should Not Talk to Kafka Directly

It is technically possible to run a Kafka producer on a gateway, but it rarely ends well for device fleets:

1. Client weight. Kafka clients need far more memory and CPU than a microcontroller has, and they assume stable connections to multiple brokers.
2. Connection model. Kafka is built for a modest number of busy producers, not tens of thousands of devices each sending a reading every few seconds over flaky links.
3. No device semantics. There is no Last Will, no keep-alive based offline detection and no retained last-known value per device.
4. Commands to devices. Kafka consumers pull; there is no natural way to push a command to one specific device behind NAT.
5. Security surface. Exposing Kafka listeners to the internet for devices means managing per-device credentials and ACLs on infrastructure designed to sit inside a private network.

Why MQTT Alone Is Not a Data Platform

MQTT has the opposite gaps. A broker routes messages; it does not keep history. If your analytics job is down for an hour, the readings it missed are gone unless a persistent session held them, and session queues are not a long-term store. You cannot replay last week's data through a new algorithm, and you cannot join streams or compute windows inside the broker. That is the work Kafka and its stream processing ecosystem are good at.

The Pattern: MQTT at the Edge, Kafka in the Core

The architecture most teams converge on looks like this:

1. Devices and gateways publish to an MQTT broker, for example connectmq/factory1/production/line1/cnc1/telemetry.
2. One or more consumer services subscribe to the relevant topic filters.
3. Those services validate, enrich and key each message, then produce it to a Kafka topic such as iot.telemetry, typically keyed by device ID to keep per-device ordering within a partition.
4. Stream processors, sinks and analytics read from Kafka.
5. Commands flow the other way: a service consumes a Kafka command topic and publishes to the device's MQTT command topic.

ConnectMQ does not include a built-in Kafka bridge, so the link between the two is always a component you run: either your own consumer service, as shown below, or an off-the-shelf MQTT source connector for Kafka Connect configured to subscribe to your broker as a normal MQTT client. Both approaches treat the broker as a standard MQTT endpoint, which keeps the integration portable.

Build an MQTT to Kafka Consumer Service

The following Python service subscribes to ConnectMQ with a shared subscription, so you can run several instances and have the broker split the load between them, and produces each message to Kafka with confluent-kafka. Shared subscriptions require MQTT 5 and are available on the paid Serverless plan and above.

pip install "paho-mqtt>=2.0" confluent-kafka

import json
import ssl
import signal
import sys

import paho.mqtt.client as mqtt
from confluent_kafka import Producer

MQTT_HOST = "broker.connectmq.com"
MQTT_FILTER = "$share/kafka-bridge/connectmq/factory1/+/+/+/telemetry"
KAFKA_TOPIC = "iot.telemetry"

producer = Producer({
    "bootstrap.servers": "localhost:9092",
    "enable.idempotence": True,
    "linger.ms": 20,
})

def delivery_report(err, record):
    if err is not None:
        print(f"Kafka delivery failed for key {record.key()}: {err}")

def on_connect(client, userdata, flags, reason_code, properties):
    print("MQTT connected:", reason_code)
    client.subscribe(MQTT_FILTER, qos=1)

def on_message(client, userdata, msg):
    try:
        payload = json.loads(msg.payload)
    except json.JSONDecodeError:
        print("skipping non-JSON payload on", msg.topic)
        return
    levels = msg.topic.split("/")
    device_id = levels[-2]  # e.g. cnc1
    record = {"topic": msg.topic, "device": device_id, "data": payload}
    producer.produce(KAFKA_TOPIC, key=device_id,
                     value=json.dumps(record), callback=delivery_report)
    producer.poll(0)

client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2,
                     client_id="YOUR_CLIENT_ID", protocol=mqtt.MQTTv5)
client.username_pw_set("YOUR_USERNAME", "YOUR_PASSWORD")
client.tls_set(cert_reqs=ssl.CERT_REQUIRED)
client.on_connect = on_connect
client.on_message = on_message

def shutdown(*_):
    client.disconnect()
    producer.flush(10)
    sys.exit(0)

signal.signal(signal.SIGINT, shutdown)
signal.signal(signal.SIGTERM, shutdown)

client.connect(MQTT_HOST, 8883, keepalive=30)
client.loop_forever()

A few production details matter more than the code itself:

• Back-pressure. If Kafka is unavailable, produce will eventually raise BufferError when the local queue fills. Decide whether to block, drop QoS 0 data or stop acknowledging MQTT messages. paho acknowledges QoS 1 messages automatically after the callback returns, so for stronger guarantees use manual acknowledgment (client.manual_ack_set(True) and client.ack(msg.mid, msg.qos) after the Kafka delivery callback).
• Keying. Use the device ID as the Kafka key so each device's readings stay ordered within one partition.
• Idempotence. QoS 1 can deliver duplicates. Include a message ID or timestamp in the payload so downstream consumers can deduplicate.
• Scaling. Add instances of the service; the $share/kafka-bridge/ prefix makes the broker distribute messages among them. Give each instance its own client ID and matching credentials, since two connections with the same client ID would disconnect each other. See MQTT shared subscriptions for the details.

To prototype without credentials, connect to port 1883 with a unique client ID, drop the $share/ prefix and use a topic prefix like connectmq/demo/<random-id>/ on the public MQTT test broker. Treat that data as public.

Choosing Between Them, or Using Both

Use MQTT only when you need real-time device connectivity, dashboards, alerts and command and control, and your history needs are covered by a time-series database fed by a subscriber. Many projects never need Kafka.

Use Kafka only when all producers are backend services on reliable networks and you need durable, replayable event streams between systems.

Use both when you have a device fleet and a data platform that needs replay, multiple independent consumers at different speeds, or stream processing at scale. Keep the boundary clean: MQTT topics model devices and assets, Kafka topics model business event streams.

For the device side, a hosted MQTT broker saves you from running two distributed systems yourself. Compare the options in the cloud MQTT broker guide, read MQTT vs AMQP if queues are also on your list, and brush up on fundamentals in the MQTT tutorial.

Connect Your Devices to ConnectMQ

Put a managed broker in front of your Kafka cluster and let each system do what it does best. Create a free Serverless broker at app.connectmq.com, no credit card required, connect a few devices, then point your consumer service at it and watch records arrive in Kafka.

Frequently Asked Questions

Can Kafka replace an MQTT broker?

Not for device fleets. Kafka lacks lightweight clients, keep-alive based offline detection, Last Will and retained state, and it is not designed for huge numbers of devices on unreliable networks.

Can MQTT replace Kafka?

Only if you do not need durable history, replay or stream processing. An MQTT broker routes messages in real time and keeps just the last retained value per topic.

How do I get MQTT data into Kafka?

Run a component that subscribes to the broker as an MQTT client and produces to Kafka, either your own consumer service or a Kafka Connect MQTT source connector. ConnectMQ does not have a built-in Kafka bridge.

Should I use QoS 1 or QoS 0 for the MQTT to Kafka bridge?

Use QoS 1 for data you cannot afford to lose, and make downstream processing tolerant of duplicates. QoS 0 is fine for high-rate telemetry where an occasional gap is acceptable.