Paho MQTT Reconnect: Robust Backoff, Resubscribe and Offline Buffering in Python

Paho MQTT reconnect behaviour is the difference between a client that survives a flaky network and one that silently stops working at 3 a.m. This guide explains what paho-mqtt 2.x does automatically, the three gaps it leaves open, and gives you a tested Python client with exponential backoff, resubscription on connect, an offline buffer and persistent MQTT 5 sessions.

Symptoms of Broken Reconnect Logic

Reconnect bugs rarely crash a program. They show up as quieter failures that are easy to misdiagnose:

• The client reconnects after a Wi-Fi drop, logs "connected", but never receives another message.
• A script started while the network was down exits immediately with a socket error instead of waiting.
• Readings taken during a short outage vanish, leaving gaps in charts.
• Hundreds of devices reconnect at exactly the same moment after a broker restart and overload the network.
• Two processes keep disconnecting each other every few seconds.

Each of these has a specific cause and a specific fix.

What Paho Does Automatically

When you run the network loop with loop_start() or loop_forever(), paho-mqtt reconnects on its own after an established connection drops. The delay between attempts starts at min_delay and doubles after every failure until it reaches max_delay, configured with reconnect_delay_set(). The defaults are 1 and 120 seconds. A successful connection resets the delay.

Paho also keeps in-flight QoS 1 and QoS 2 messages that the broker has not yet acknowledged and retransmits them after reconnecting, as long as the session is resumed.

That covers a lot, but not everything. The remaining gaps are your responsibility.

Gap 1: The First Connection Fails

client.connect() performs the first connection synchronously and raises an exception, typically ConnectionRefusedError, socket.gaierror or TimeoutError, if the broker is unreachable. Automatic reconnection never starts because there was never a connection. This is why scripts launched at boot, before the network is ready, often die.

The fix is connect_async() followed by loop_start(). The connection attempt moves into the network loop, which retries using the same backoff rules.

Gap 2: Subscriptions Disappear

With a clean session, the broker discards your subscriptions when the connection ends. Paho does not record them and does not resubscribe for you. If you call subscribe() once after connect(), the client will reconnect happily and then receive nothing.

The fix is simple and non-negotiable: subscribe inside on_connect. That callback runs after every successful connection, including reconnects, so subscriptions are always restored.

Gap 3: Messages Published While Offline

Readings produced while the link is down need somewhere to go. QoS 0 messages are fire-and-forget, so publishing while disconnected drops them. Relying on paho's internal queue for higher QoS levels is not a good plan either, because it is unbounded and invisible to your application. An explicit, bounded offline buffer that you flush after reconnecting gives you control over memory and over what to discard when an outage lasts too long.

A Robust Paho MQTT Reconnect Client

The class below closes all three gaps and adds jitter to the backoff so a fleet of devices does not reconnect in lockstep. It uses MQTT 5 with a session expiry so the broker keeps subscriptions and queued QoS 1 messages during short outages.

pip install "paho-mqtt>=2.0"

import json
import logging
import random
import threading
import time
import uuid
from collections import deque

import paho.mqtt.client as mqtt
from paho.mqtt.packettypes import PacketTypes
from paho.mqtt.properties import Properties

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("mqtt")


class ResilientClient:
    def __init__(self, host, port, client_id, subscriptions,
                 username=None, password=None, use_tls=False,
                 session_expiry_s=1800, buffer_size=1000):
        self.host = host
        self.port = port
        self.subscriptions = subscriptions  # list of (topic, qos)
        self.session_expiry_s = session_expiry_s
        self.offline = deque(maxlen=buffer_size)  # oldest dropped when full
        self.lock = threading.Lock()
        self.on_payload = None

        c = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2,
                        client_id=client_id, protocol=mqtt.MQTTv5)
        if username:
            c.username_pw_set(username, password)
        if use_tls:
            c.tls_set()
        # Exponential backoff; a random floor spreads out reconnecting fleets
        c.reconnect_delay_set(min_delay=random.randint(1, 5), max_delay=60)
        c.on_connect = self._on_connect
        c.on_disconnect = self._on_disconnect
        c.on_connect_fail = self._on_connect_fail
        c.on_message = self._on_message
        self.client = c

    def start(self):
        props = Properties(PacketTypes.CONNECT)
        props.SessionExpiryInterval = self.session_expiry_s
        self.client.connect_async(self.host, self.port, keepalive=30,
                                  clean_start=False, properties=props)
        self.client.loop_start()

    def stop(self):
        self.client.disconnect()
        self.client.loop_stop()

    # --- callbacks -------------------------------------------------------
    def _on_connect(self, client, userdata, flags, reason_code, properties):
        if reason_code.is_failure:
            log.warning("connect refused: %s", reason_code)
            return
        log.info("connected, session_present=%s", flags.session_present)
        if self.subscriptions:
            client.subscribe(self.subscriptions)  # always resubscribe
        self._flush()

    def _on_disconnect(self, client, userdata, flags, reason_code, properties):
        if reason_code == 0:
            log.info("clean disconnect")
        else:
            log.warning("connection lost: %s, reconnecting", reason_code)

    def _on_connect_fail(self, client, userdata):
        log.warning("connection attempt failed, retrying with backoff")

    def _on_message(self, client, userdata, msg):
        if self.on_payload:
            self.on_payload(msg.topic, msg.payload)

    # --- publishing ------------------------------------------------------
    def publish(self, topic, payload, qos=1, retain=False):
        if self.client.is_connected():
            info = self.client.publish(topic, payload, qos=qos, retain=retain)
            if info.rc == mqtt.MQTT_ERR_SUCCESS:
                return True
        with self.lock:
            self.offline.append((topic, payload, qos, retain))
        return False

    def _flush(self):
        with self.lock:
            pending = list(self.offline)
            self.offline.clear()
        if pending:
            log.info("flushing %d buffered messages", len(pending))
        for topic, payload, qos, retain in pending:
            self.client.publish(topic, payload, qos=qos, retain=retain)


if __name__ == "__main__":
    run_id = uuid.uuid4().hex[:8]
    prefix = f"connectmq/demo/{run_id}"
    rc = ResilientClient(
        host="broker.connectmq.com", port=1883,
        client_id=f"resilient-{run_id}",
        subscriptions=[(f"{prefix}/cmd/#", 1)],
    )
    rc.on_payload = lambda t, p: log.info("command on %s: %s", t, p)
    rc.start()
    print(f"publishing to {prefix}/telemetry, send commands to {prefix}/cmd/...")
    try:
        n = 0
        while True:
            n += 1
            payload = json.dumps({"seq": n, "ts": int(time.time() * 1000)})
            sent = rc.publish(f"{prefix}/telemetry", payload, qos=1)
            log.info("seq %d %s (buffered: %d)", n, "sent" if sent else "buffered", len(rc.offline))
            time.sleep(2)
    except KeyboardInterrupt:
        rc.stop()

Why each piece is there

The sequence number in each payload makes testing easy: after an outage, gaps or duplicates in seq show exactly what happened. Because QoS 1 is at-least-once delivery, consumers should tolerate an occasional duplicate, for example by ignoring sequence numbers they have already seen. Our explainer on MQTT QoS levels covers why duplicates are possible.

How to Test Reconnect Logic

Do not trust reconnect code you have not broken on purpose. Run the script and a subscriber side by side:

mosquitto_sub -h broker.connectmq.com -p 1883 -i watch-<run-id> -t "connectmq/demo/<run-id>/#" -v

Then try each failure mode in turn:

1. Network drop. Disable Wi-Fi or unplug the cable for 30 seconds. The log shows "connection lost", then failing attempts with growing delays, then "connected" and "flushing N buffered messages". The subscriber sees the buffered sequence numbers arrive in a burst.
2. Blocked port on Linux. Simulate a broker that is unreachable while the rest of the network works:

sudo iptables -A OUTPUT -p tcp --dport 1883 -j DROP
# wait 60 seconds, then remove the rule
sudo iptables -D OUTPUT -p tcp --dport 1883 -j DROP

Dropped packets are a harsher test than a refused connection, because the client only notices through the keep-alive timeout, roughly one and a half times the keep-alive interval. 3. Start offline. Disconnect the network before starting the script. With connect_async(), it waits and connects when the network returns instead of exiting. 4. Session takeover. Start a second client with the same client ID, for example mosquitto_sub -i resilient-<run-id> -t x. The broker disconnects the original client, which reconnects and in turn disconnects the intruder. This is the "two processes fighting" symptom; seeing it on purpose makes it easy to recognise later. 5. Commands during an outage. While the script is offline, publish a QoS 1 command to its cmd topic. Because the session was kept, the command is delivered when the client returns.

Reason Codes Worth Handling

MQTT 5 tells you why a connection ended. Log the reason code and react to the ones that retrying will not fix.

Endlessly retrying a 0x86 refusal only floods the broker's logs, and many brokers, ConnectMQ included, block an IP address after repeated failed logins. In _on_connect, consider calling client.disconnect() and alerting a human when the code indicates a permanent problem.

Production Settings

For real deployments, connect over TLS on port 8883 with credentials issued by your broker. Against a hosted MQTT broker from ConnectMQ, construct the client like this:

rc = ResilientClient(
    host="broker.connectmq.com", port=8883, use_tls=True,
    client_id="YOUR_CLIENT_ID", username="YOUR_USERNAME", password="YOUR_PASSWORD",
    subscriptions=[("connectmq/factory1/production/line1/+/cmd", 1)],
)

Keep a few limits in mind. On the public broker, sessions expire after at most 30 minutes, so the 1,800-second expiry used above is the maximum useful value there. The free Serverless plan supports QoS up to 1, which is what this client uses. Paid plans add QoS 2 if you need exactly-once delivery. The broker also publishes connection state on $SYS/broker/connection/{clientId}/state, which is handy for confirming that a device came back after an outage.

For broader background, our paho-mqtt Python tutorial covers the basics, the paho-mqtt 2.0 callback API guide explains the version 2 callback signatures used here, and the MQTT client libraries comparison helps if you are choosing between paho and async libraries. The official paho-mqtt documentation lists every reconnect-related option.

Get Started with ConnectMQ

Test your reconnect logic against the free public MQTT broker today, then move to private credentials when you deploy. ConnectMQ runs a fully managed MQTT 5 broker with TLS, session expiry and clear reason codes, so your clients get useful feedback when something goes wrong. Create a free Serverless broker at app.connectmq.com with no credit card.

Frequently Asked Questions

Does paho-mqtt reconnect automatically?

Yes, once a connection has been established and the network loop is running with loop_start() or loop_forever(). It does not retry a failed first connect() call, and it does not resubscribe for you.

Why does my paho client stop receiving messages after reconnecting?

The subscriptions were lost with the old session. Move your subscribe() calls into on_connect so they run after every successful connection.

What reconnect delay should I use?

Start between 1 and 5 seconds and cap at 60 to 120 seconds. Randomising the starting delay per device prevents a whole fleet from reconnecting at the same instant.

Should I use clean_start=False?

Use it with a session expiry interval when you want the broker to keep subscriptions and queue QoS 1 messages during short outages. Keep the client ID stable, because the session is tied to it.