Designing IoT Clients to Survive an MQTT Broker Outage

An MQTT broker outage is not a question of if but when: networks partition, certificates rotate, brokers are upgraded and cloud regions have bad days. This guide shows you how to design device and backend clients so that a broker outage, short or long, loses no important data and causes no duplicate actions. The patterns apply to any broker, including ConnectMQ, and come with working Python code.

What Actually Happens During an MQTT Broker Outage

From the client's point of view, an outage looks like one of three things: the TCP connection drops, the connection hangs silently until the keepalive expires, or new connection attempts are refused. The cause may be the broker itself, the network between you and the broker, DNS, TLS, or even your own credentials. The client cannot always tell which, so it must handle all of them the same way: notice quickly, keep working locally and reconnect politely.

Four things are at risk:

1. Outgoing data published while disconnected is lost unless you buffer it.
2. Incoming commands sent while you were offline may or may not be delivered, depending on session settings.
3. Presence information becomes wrong if other systems still think the device is online.
4. The broker itself can be overwhelmed when thousands of devices reconnect at the same instant.

Good client design addresses each one. It is worth reading what an MQTT broker does with sessions and queued messages if those terms are unfamiliar.

Pattern 1: Reconnect with Exponential Backoff and Jitter

The naive approach is to retry every second forever. Multiply that by ten thousand devices and the moment the broker comes back it receives a wall of simultaneous connections, which can knock it straight over again. This is the thundering herd problem.

The fix is exponential backoff with random jitter: wait 1 s, then 2 s, 4 s, 8 s and so on up to a cap, and randomise each delay so devices spread out. Paho 2.x has built-in support via reconnect_delay_set, but it does not add jitter, so many teams manage the loop themselves for full control.

Also mind your credentials. Brokers that block IP addresses after repeated failed logins, as ConnectMQ does, will treat a device with a wrong password retrying every second as an attack. Back off on authentication errors much more aggressively than on network errors, and surface them to an operator.

Pattern 2: Buffer Outgoing Data Locally

MQTT client libraries keep in-flight messages in memory, which disappears if the device reboots during the outage. For telemetry you cannot afford to lose, write each reading to a small local store first and delete it only after the broker acknowledges it with a PUBACK. SQLite is a good fit on gateways and Linux devices; on microcontrollers, a ring buffer in flash serves the same purpose.

Decide up front what happens when the buffer fills. For most telemetry, dropping the oldest readings is correct. For billing or safety events, you may prefer to stop sampling non-critical data instead.

Pattern 3: Announce Presence with Last Will and Retained Status

Set a Last Will and Testament (LWT) when connecting: a retained offline message the broker publishes on your behalf if you disappear without a clean disconnect. On every successful connect, publish a retained online to the same topic. Other systems then see accurate presence without polling.

Be precise about LWT semantics. The will is published when the broker detects that the session ended abnormally, usually after 1.5 times the keepalive interval. If the broker itself is down, nothing is published until it returns, and after a broker restart the retained state may need to be re-established by the clients. MQTT 5 adds a will delay interval, which lets you suppress flapping offline messages during a brief reconnect.

Pattern 4: Make Commands Idempotent

With QoS 1, a message is delivered at least once. During an outage and reconnect, duplicates become likely. A "toggle relay" command delivered twice turns the relay back off; "dispense 10 ml" delivered twice dispenses 20 ml. The remedy is to make every command idempotent:

• Send absolute states ({"relay": "on"}) rather than relative actions (toggle).
• Include a unique command_id in each command and have the device remember recently executed IDs.
• Include an expiry: an MQTT 5 message expiry interval, or a valid_until timestamp, so a command queued during a long outage is not executed hours later when it is no longer safe.

A Resilient Python Client

The following client combines the four patterns. It buffers telemetry in SQLite, reconnects with randomised exponential backoff, maintains a retained status topic with an LWT and de-duplicates commands. It uses the public MQTT test broker so you can run it immediately; switch to port 8883, TLS and your portal credentials for production.

import json
import random
import sqlite3
import time
import uuid

import paho.mqtt.client as mqtt

HOST, PORT = "broker.connectmq.com", 1883
DEVICE = f"dev-{uuid.uuid4().hex[:6]}"
ROOT = f"connectmq/demo/{DEVICE}"
STATUS, TELEMETRY, COMMANDS = f"{ROOT}/status", f"{ROOT}/telemetry", f"{ROOT}/cmd"

db = sqlite3.connect("outbox.db", check_same_thread=False)
db.execute("CREATE TABLE IF NOT EXISTS outbox (id INTEGER PRIMARY KEY, payload TEXT)")
db.commit()

pending = {}            # mid -> outbox row id
seen_commands = []      # recent command IDs for de-duplication
connected = False


def on_connect(client, userdata, flags, reason_code, properties):
    global connected
    if reason_code.is_failure:
        print("Connect refused:", reason_code)
        return
    connected = True
    print("Connected")
    client.publish(STATUS, "online", qos=1, retain=True)
    client.subscribe(COMMANDS, qos=1)


def on_disconnect(client, userdata, flags, reason_code, properties):
    global connected
    connected = False
    pending.clear()  # unacknowledged rows stay in the outbox and are resent
    print("Disconnected:", reason_code)


def on_publish(client, userdata, mid, reason_code, properties):
    row_id = pending.pop(mid, None)
    if row_id is not None:
        db.execute("DELETE FROM outbox WHERE id = ?", (row_id,))
        db.commit()


def on_message(client, userdata, msg):
    try:
        cmd = json.loads(msg.payload)
    except ValueError:
        print("Ignoring malformed command")
        return
    if cmd.get("command_id") in seen_commands:
        print("Duplicate command ignored:", cmd["command_id"])
        return
    if cmd.get("valid_until", float("inf")) < time.time():
        print("Expired command ignored:", cmd.get("command_id"))
        return
    seen_commands.append(cmd.get("command_id"))
    del seen_commands[:-100]
    print("Executing:", cmd)


def flush_outbox(client):
    rows = db.execute("SELECT id, payload FROM outbox ORDER BY id LIMIT 20").fetchall()
    for row_id, payload in rows:
        if row_id in pending.values():
            continue
        info = client.publish(TELEMETRY, payload, qos=1)
        pending[info.mid] = row_id


client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id=DEVICE)
client.will_set(STATUS, "offline", qos=1, retain=True)
client.on_connect, client.on_disconnect = on_connect, on_disconnect
client.on_publish, client.on_message = on_publish, on_message

# Exponential backoff between reconnects; a random minimum spreads devices out
client.reconnect_delay_set(min_delay=random.randint(1, 5), max_delay=120)
client.connect_async(HOST, PORT, keepalive=30)  # retried by the loop if it fails
client.loop_start()

while True:
    reading = {"temperature": round(random.uniform(20, 25), 2), "ts": int(time.time())}
    db.execute("INSERT INTO outbox (payload) VALUES (?)", (json.dumps(reading),))
    db.commit()
    if connected:
        flush_outbox(client)
    time.sleep(5)

Test it by disconnecting your network for a minute. Readings accumulate in outbox.db, the network loop keeps retrying with growing delays, and once the network returns the backlog drains in order, up to twenty rows per cycle. Paho doubles the delay between attempts up to max_delay but adds no jitter of its own, which is why each device picks a random starting delay. On a large fleet you can go further and add random jitter to every attempt with your own reconnect loop.

Session State and What Not to Assume

Persistent sessions (clean session false in MQTT 3.1.1, or a session expiry interval in MQTT 5) let the broker queue QoS 1 and 2 messages for a disconnected client. They help during client-side outages. During broker-side outages they help only if the broker persists sessions across restarts, which not every broker does. ConnectMQ does not guarantee persistence across restarts, and on the public broker sessions expire after at most 30 minutes.

So design defensively:

1. Re-subscribe in on_connect every time, rather than relying on the broker to remember subscriptions.
2. Re-publish retained status and configuration after reconnecting.
3. Treat device-side buffering, not broker queues, as the source of truth for data you cannot lose.
4. Let the backend reconcile state after a gap, for example by asking devices to republish their current configuration.

Outage Readiness Checklist

For broker-side measures that complement these client patterns, see our guides on MQTT broker high availability and MQTT broker monitoring, and run through the MQTT production checklist before launch.

Run Resilient Clients on ConnectMQ

A managed MQTT broker removes the most common self-inflicted outages, such as expired certificates and unpatched servers, while your clients handle the rest. Create a free broker at app.connectmq.com on the Serverless plan with no credit card, test these patterns against it, and use the portal's $SYS connection state and alerts log to watch reconnects. Read more about ConnectMQ or compare options in our cloud MQTT broker guide.

Frequently Asked Questions

How fast should an IoT client reconnect after an MQTT broker outage?

Try quickly at first, then back off exponentially to a cap of one to two minutes, with random jitter on every delay. This recovers fast from blips without overwhelming the broker after a long outage.

Does QoS 1 protect data during a broker outage?

Only for messages already handed to the client library, and only while the device stays powered. For data you cannot lose, buffer it in persistent local storage until a PUBACK arrives.

When is the Last Will message sent?

When the broker detects an abnormal disconnect, typically after about 1.5 times the keepalive interval. It is not sent on a clean disconnect, and nothing is sent while the broker itself is down.

Should commands use QoS 2 to avoid duplicates?

QoS 2 avoids duplicates between client and broker, but end-to-end duplicates can still occur after reconnects or application retries. Idempotent commands with unique IDs are the more robust fix.