Handling Duplicate MQTT Messages with QoS 1

MQTT duplicate messages are not a bug in your broker; they are a documented consequence of QoS 1, which promises delivery "at least once". When a network hiccup eats an acknowledgement, the sender retransmits, and your consumer sees the same message twice. In this troubleshooting guide you will learn exactly when duplicates happen, what the DUP flag does and does not tell you, and how to write an idempotent Python consumer that handles them correctly.

Why QoS 1 Produces Duplicates

QoS 1 uses a two-step handshake on each hop: the sender transmits PUBLISH, the receiver replies with PUBACK. Until the PUBACK arrives, the sender keeps the message and will resend it. The MQTT QoS levels explained article covers all three levels; here we focus on what goes wrong with level 1.

A duplicate is created whenever the receiver got the message but the sender did not get the acknowledgement:

1. The device publishes reading #42 to the broker.
2. The broker receives it, forwards it to subscribers, and sends PUBACK.
3. The connection drops before PUBACK reaches the device.
4. The device reconnects with its session and resends #42 with the DUP flag set.
5. The broker treats it as a new message and forwards it again.

The same thing happens on the broker-to-subscriber hop. If the subscriber processed a message but its PUBACK was lost, the broker resends it when the subscriber's session resumes.

Note that a duplicate can arrive on either hop independently, so one published message can occasionally reach a consumer more than twice.

What the DUP Flag Tells You

Every PUBLISH packet has a DUP bit. A sender sets it when it is retransmitting a QoS 1 or QoS 2 packet. It is tempting to drop every message with DUP set, but that is wrong for several reasons:

The DUP flag is useful for logging and diagnostics. It is not a deduplication key. Real deduplication needs an identifier that you put into the message yourself.

Reproducing Duplicates

Real redelivery depends on lost packets and is hard to trigger on demand. The reliable way to test your consumer is to inject duplicates yourself. Against the public MQTT test broker at broker.connectmq.com, publish the same message ID twice:

for n in 1 2; do
  mosquitto_pub -h broker.connectmq.com -p 1883 -i dup-pub-t9v4 -q 1 \
    -t connectmq/demo/t9v4/meter-3/energy \
    -m '{"msgId":"meter-3-000042","kwh":0.25}'
done

A consumer that sums kwh without deduplication now reports 0.5 instead of 0.25. That is exactly the class of bug that QoS 1 redelivery causes in production, only rarer and harder to spot.

Building an Idempotent Consumer

An idempotent consumer produces the same result whether it processes a message once or five times. There are two common techniques:

1. Natural idempotency. Design the effect so repetition is harmless: "set temperature to 21.5" instead of "increase by 0.5"; upsert a row keyed by device and timestamp instead of inserting.
2. Explicit deduplication. Give each message a unique ID and remember recently processed IDs.

Here is a complete paho-mqtt 2.x consumer that uses explicit deduplication with a bounded, time-limited cache:

import json
import time
import uuid
from collections import OrderedDict
import paho.mqtt.client as mqtt

BROKER = "broker.connectmq.com"
TOPIC = "connectmq/demo/t9v4/+/energy"
DEDUP_TTL = 600          # seconds to remember an ID
DEDUP_MAX = 50_000       # cap memory use

seen = OrderedDict()     # msgId -> first-seen time
totals = {}

def already_seen(msg_id):
    now = time.time()
    while seen:
        oldest_id, ts = next(iter(seen.items()))
        if now - ts > DEDUP_TTL or len(seen) > DEDUP_MAX:
            seen.popitem(last=False)
        else:
            break
    if msg_id in seen:
        return True
    seen[msg_id] = now
    return False

def on_connect(client, userdata, flags, reason_code, properties):
    client.subscribe(TOPIC, qos=1)

def on_message(client, userdata, msg):
    try:
        data = json.loads(msg.payload)
        msg_id = data["msgId"]
    except (ValueError, KeyError):
        print("rejected message without msgId")
        return
    if already_seen(msg_id):
        print(f"duplicate {msg_id} (dup flag={msg.dup}) ignored")
        return
    meter = msg.topic.split("/")[-2]
    totals[meter] = totals.get(meter, 0) + data["kwh"]
    print(f"{meter} total={totals[meter]:.2f} kWh")

client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2,
                     client_id=f"dedup-{uuid.uuid4().hex[:8]}")
client.on_connect = on_connect
client.on_message = on_message
client.connect(BROKER, 1883, keepalive=30)
client.loop_forever()

Run it, then run the double-publish loop from the previous section. The first message updates the total; the second is reported as a duplicate and ignored.

Choosing a Good Message ID

The ID must be created by the original publisher, once, and reused on every retry. Options:

• Device ID plus sequence number, such as meter-3-000042. Compact and also useful for MQTT message ordering checks. Persist the counter or add a boot ID so it does not repeat after a reboot.
• UUID generated per message. Simple and globally unique, at the cost of 36 bytes.
• Device ID plus measurement timestamp, if the device never produces two readings with the same timestamp.

Do not generate the ID in the consumer or in a middle layer; by then a duplicate looks like a new message.

Scaling Deduplication

The in-memory cache above works for a single consumer process. For larger systems:

• Database constraints. A unique index on msg_id turns duplicates into harmless conflicts. With an upsert (INSERT ... ON CONFLICT DO NOTHING in PostgreSQL) this is often the only deduplication you need.
• Shared cache. If several consumers in a shared subscription can receive retries of the same message, the seen-set must be shared, for example in Redis with a TTL.
• TTL sizing. The window only needs to cover realistic redelivery delays: your session expiry plus a margin. Duplicates rarely arrive hours apart unless sessions last that long.

Should You Just Use QoS 2?

QoS 2 adds a four-step handshake that guarantees exactly-once delivery on each hop. It costs more round trips and broker state, and it does not protect you against a device that times out and republishes the same reading as a new message at the application level. On constrained plans it may not be available at all; for instance, the free ConnectMQ Serverless plan supports QoS up to 1 while the paid Serverless plan adds QoS 2. See MQTT maximum QoS for how brokers advertise and downgrade QoS. In most systems QoS 1 plus idempotent consumers is the more robust and cheaper design.

Troubleshooting Checklist

1. Are duplicates correlated with reconnects? Check client logs for disconnects and keepalive timeouts.
2. Does your publisher retry at the application level on top of QoS 1? That doubles the retry paths.
3. Do two clients share a client ID and kick each other off, causing constant session resumption? See MQTT client keeps disconnecting.
4. Is the consumer slow to acknowledge? Long processing inside the callback delays PUBACK and widens the window for redelivery.
5. Does every message carry an ID created at the source?

Duplicates and Your Broker Choice

ConnectMQ implements standard QoS 1 and QoS 2 semantics, so the advice above applies directly. Because client IDs are bound to issued credentials, the duplicate-client-ID reconnect loop that often causes mass redelivery is easier to avoid. A managed MQTT broker handles session storage and retries for you; idempotency in your consumers is still your responsibility, on any broker. The MQTT tutorial is a good next step for the overall picture.

Get Started with ConnectMQ

Create a free broker at app.connectmq.com on the Serverless plan, no credit card required. Point the deduplicating consumer at it with your ConnectMQ username, password and client ID, and your totals stay correct even when the network does not.

Frequently Asked Questions

Why am I receiving the same MQTT message twice?

With QoS 1, a lost acknowledgement causes the sender to resend. The receiver gets the message again, which is allowed by the at-least-once guarantee.

Can I rely on the DUP flag to drop duplicates?

No. DUP marks a retransmission on one hop, but the first copy may never have arrived, and the broker does not carry the flag through to subscribers.

Does QoS 0 produce duplicates?

No, QoS 0 is never retransmitted by the protocol. It can lose messages instead.

How long should I remember message IDs?

Long enough to cover reconnect and session resumption delays, typically minutes. Size it from your session expiry interval plus a safety margin.