How to Rotate MQTT Credentials Without Downtime

To rotate MQTT credentials without downtime you need two credential sets valid at the same time, devices that can fall back from the new one to the old one, and a verification step before anything is revoked. This guide gives you that dual-credential strategy step by step, including device-side code and the order of operations for issuing and revoking credentials on a managed broker.

Why Rotate MQTT Credentials at All?

Static credentials have a long shelf life on devices. Over the years they pass through build pipelines, support laptops, returned hardware and contractor handovers. Rotation limits how long any leaked copy stays useful. You also need a tested rotation process for unplanned events:

A device is stolen, returned or decommissioned.
A firmware image with embedded secrets was published by mistake.
An employee or integrator with access to credentials leaves.
A security review or customer contract requires periodic rotation.

The worst time to design a rotation process is during an incident. Running it on a schedule keeps it working.

Why Naive Rotation Causes Outages

The tempting approach is: change the password on the broker, then push the new one to devices. Every device that has not received the update yet is locked out, and a device that is locked out of MQTT often cannot receive the update either, because the update channel is MQTT. You end up with a field visit for each offline device.

The reverse order fails too: if devices switch first and the broker does not know the new credential yet, they cannot connect. The only safe approach is an overlap window where both credentials work.

The Dual-Credential Rotation Strategy

Phase	Broker state	Device state	Exit condition
1. Issue	Old and new both valid	Old only	New credential created and recorded
2. Stage	Old and new both valid	Old active, new stored	Device confirms new credential stored
3. Switch	Old and new both valid	New active, old as fallback	Device connected with new credential
4. Verify	Old and new both valid	New active	No connections with old credential for a set period
5. Revoke	New only	New only, old deleted	Old credential removed everywhere

The phases apply to any broker. What changes is how credentials are issued and revoked.

Phase 1: Issue the New Credential

On a self-hosted Mosquitto broker this means adding a second user to the password file. On ConnectMQ, issue a new credential set for the device in the portal. Each set comes with its own username, password and client ID, because client IDs are bound to the issued credentials. Record the mapping from physical device to old and new client IDs in your inventory before going further.

That binding has a practical consequence: during rotation the device does not just change its password, it changes its whole identity triple. Make sure any consumers that key on client ID, such as presence monitoring on $SYS/broker/connection/{clientId}/state or your own inventory, know about both IDs during the overlap.

Phase 2: Stage It on the Device

Deliver the new credential over the existing, authenticated, TLS-protected connection. A typical pattern is a command topic only that device listens to:

devices/<deviceId>/cmd/credentials    # backend -> device, QoS 1, not retained
devices/<deviceId>/evt/credentials    # device -> backend, acknowledgement

Never publish credentials as retained messages, and never to a shared or wildcard-readable topic. If your broker does not enforce per-topic permissions, keep untrusted clients off it entirely, or deliver credentials through a separate channel such as your OTA update service. The device writes the new credential to protected storage in a separate slot from the current one and acknowledges with a hash, not the password itself.

Phase 3: Switch With Fallback

The device reconnects with the new credential. If the broker rejects it, the device falls back to the old one and reports the failure. This is the step that makes the rotation safe.

import json
import os
import ssl
import time
import paho.mqtt.client as mqtt

HOST = "YOUR_BROKER_HOST"
STORE = "/var/lib/mydevice/credentials.json"   # protect with 0600 permissions

def load():
    with open(STORE) as f:
        return json.load(f)   # {"active": {...}, "pending": {...} or null}

def save(data):
    tmp = STORE + ".tmp"
    with open(tmp, "w") as f:
        json.dump(data, f)
    os.replace(tmp, STORE)    # atomic swap, survives power loss mid-write

def try_connect(cred, timeout=15):
    result = {}
    def on_connect(client, userdata, flags, reason_code, properties):
        result["rc"] = reason_code
    c = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2,
                    client_id=cred["client_id"], protocol=mqtt.MQTTv5)
    c.username_pw_set(cred["username"], cred["password"])
    c.tls_set(cert_reqs=ssl.CERT_REQUIRED)
    c.on_connect = on_connect
    c.connect(HOST, 8883, keepalive=60)
    c.loop_start()
    deadline = time.time() + timeout
    while "rc" not in result and time.time() < deadline:
        time.sleep(0.2)
    if result.get("rc") is not None and not result["rc"].is_failure:
        return c
    c.loop_stop()
    c.disconnect()
    return None

def connect_with_rotation():
    data = load()
    if data.get("pending"):
        client = try_connect(data["pending"])
        if client:
            data["previous"] = data["active"]   # keep as fallback until revoked
            data["active"], data["pending"] = data["pending"], None
            save(data)
            return client
        print("New credential rejected, falling back")
    client = try_connect(data["active"])
    if client is None and data.get("previous"):
        client = try_connect(data["previous"])
    return client

Keep the old credential as previous until the backend tells the device it has been revoked. Then delete it from storage.

Phase 4: Verify

Before revoking anything, confirm the fleet has moved. Useful signals:

Each device's acknowledgement on its evt/credentials topic.
Presence of the new client ID on $SYS/broker/connection/{clientId}/state and absence of the old one.
Your inventory report: devices still on the old credential after the window.

Investigate the stragglers individually. Offline devices are common; they will switch the next time they come online, as long as you do not revoke too early.

Phase 5: Revoke

Remove the old credential set in the portal (or delete the user and reload Mosquitto). Then tell devices to delete the previous slot. From this point a leaked copy of the old credential is useless.

Rotation Checklist

Inventory maps every device to its current client ID.
New credential issued and recorded before any device changes.
Credentials delivered only over TLS, to a device-specific topic or channel, never retained.
Device stores the new credential in a separate slot with an atomic write.
Device falls back to the old credential on rejection and reports it.
Overlap window long enough for devices that are offline for days.
Revocation only after verification, and device-side deletion after revocation.
Run the process on a schedule, not only during incidents.

Pitfalls to Watch

Tight retry loops with a wrong password. ConnectMQ blocks IP addresses after repeated failed logins. A whole site behind one NAT address can be blocked by a few misbehaving devices. Back off exponentially and fall back quickly.
Two devices with one credential. Because the client ID is bound to the credential, two devices sharing a set will take over each other's session and disconnect in a loop. Every device needs its own set.
Connection quota during overlap. If devices briefly hold both connections, or you test new credentials from a laptop, count them against your plan's connection limit.
Forgetting non-device clients. Node-RED flows, Telegraf agents and backend services hold credentials too. Rotate them on the same schedule.

Storage on the device matters as much as the rotation process. See secure MQTT credentials on devices for ESP32 and Linux options, and secure IoT device provisioning for getting the first credential onto the device safely.

Get Started with ConnectMQ

A managed MQTT broker takes the password files and certificate renewals off your plate so you can focus on the device side of rotation. Create a free Serverless broker at app.connectmq.com, no credit card required, and test the full dual-credential flow with a couple of devices before rolling it out. For background on how credentials travel in CONNECT, read MQTT username and password authentication, and to see how this fits a production setup, read about running a cloud MQTT broker.

Frequently Asked Questions

How often should I rotate MQTT credentials?

There is no universal interval. Choose one your process can sustain, commonly yearly or at each major firmware release, and always rotate immediately after a suspected leak or a device leaving your control.

Can I rotate credentials on devices that are offline?

Yes, as long as the old credential stays valid until they come back. The device switches on its next connection, and you revoke only after it has.

Why does the client ID change during rotation on ConnectMQ?

ConnectMQ binds client IDs to issued credentials, so a new credential set comes with its own client ID. Update any presence monitoring or inventory that keys on the client ID.

What if the new credential is rejected?

The device should fall back to the old credential, reconnect and report the failure so you can fix the issued credential before trying again.