MQTT Shared Subscriptions: Load Balancing Message Consumers

MQTT shared subscriptions let several clients subscribe as a group so that each message on a topic is delivered to only one member of that group, instead of to all of them. By the end of this article you will understand the $share/group/topic syntax, how delivery works when workers come and go, how to scale backend consumers with it, and how to check whether the broker you use supports it.

Why Normal Subscriptions Do Not Scale Consumers

In standard publish/subscribe, every subscriber whose filter matches receives its own copy of each message. That is exactly what you want for dashboards: every screen shows every reading. It is exactly what you do not want for backend processing. If you run three instances of a service that writes telemetry to a database, three normal subscriptions mean every reading is written three times.

Before MQTT 5, the workarounds were awkward: a single consumer that became a bottleneck, partitioning topics by hand across instances, or bridging MQTT into a separate queueing system. Shared subscriptions solve the problem inside the MQTT broker itself.

How MQTT Shared Subscriptions Work

A shared subscription uses a special filter format, standardised in MQTT 5:

$share/<ShareName>/<TopicFilter>

• $share is the fixed prefix that marks the subscription as shared.
• <ShareName> names the consumer group. It must not contain /, + or #.
• <TopicFilter> is a normal filter, wildcards included.

Every client that subscribes to the same $share/<ShareName>/<TopicFilter> joins the same group. For each matching message, the broker picks one member of the group and delivers the message to it. Different share names are independent groups, and normal subscribers to the plain topic still receive their own copies.

The specification leaves the choice of member to the broker. Round-robin, random and "least busy" strategies are all common, so do not rely on a specific order or on messages from one device always reaching the same worker.

Delivery Semantics You Need to Know

1. Ordering is per member, not per topic. Each worker sees its own messages in order, but two consecutive readings from the same sensor may go to different workers and be processed in parallel. If order matters, include a sequence number or timestamp and resolve it in your database.
2. QoS still applies. With QoS 1, if a worker disconnects before acknowledging a message, brokers typically redeliver it to the same session when it reconnects, or route it to another member; behaviour varies by broker. Make handlers idempotent.
3. Retained messages are not sent on subscribe. The specification says retained messages are not delivered to a shared subscription when it is created, because it would be unclear which member should receive them.
4. No Local is forbidden. Setting the No Local subscription option on a shared subscription is a protocol error.
5. Sessions are per member. Each worker has its own client ID and session. A persistent session on one worker keeps that worker's subscription alive while it is offline. See MQTT persistent sessions for how that affects queuing.

Example: Scaling Python Workers

The worker below joins a group called db-writers and processes telemetry. Run it in two or three terminals with different worker numbers to watch the load spread.

import sys
import json
import paho.mqtt.client as mqtt

WORKER = sys.argv[1] if len(sys.argv) > 1 else "1"
FILTER = "$share/db-writers/connectmq/demo/s7h3/factory/+/telemetry"

def on_connect(client, userdata, flags, reason_code, properties):
    print(f"Worker {WORKER} connected: {reason_code}")
    client.subscribe(FILTER, qos=1)

def on_subscribe(client, userdata, mid, reason_code_list, properties):
    for rc in reason_code_list:
        if rc.is_failure:
            print(f"Subscribe rejected: {rc}")  # e.g. Shared Subscriptions not supported
        else:
            print(f"Joined group, granted {rc}")

def on_message(client, userdata, msg):
    data = json.loads(msg.payload)
    print(f"Worker {WORKER} got {msg.topic}: {data}")
    # write to your database here; keep it idempotent

client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2,
                     client_id=f"db-writer-s7h3-{WORKER}",
                     protocol=mqtt.MQTTv5)
client.on_connect = on_connect
client.on_subscribe = on_subscribe
client.on_message = on_message
client.connect("broker.connectmq.com", 1883, keepalive=60)
client.loop_forever()

Start two workers:

python worker.py 1
python worker.py 2

Then publish a stream of test readings:

for i in $(seq 1 10); do
  mosquitto_pub -h broker.connectmq.com -p 1883 -V mqttv5 -i sh-pub-s7h3 -q 1 \
    -t "connectmq/demo/s7h3/factory/line$((i % 3))/telemetry" \
    -m "{\"seq\": $i, \"temp\": 70}"
done

If the broker supports shared subscriptions, the ten messages are split between the workers instead of each worker printing all ten. Replace s7h3 with your own random string; the public MQTT test broker is readable by anyone, so publish only dummy data.

The same subscription in MQTT.js looks like this:

const mqtt = require("mqtt");
const client = mqtt.connect("mqtts://broker.connectmq.com:8883", {
  clientId: "db-writer-node-s7h3-1",
  protocolVersion: 5,
});
client.on("connect", () => {
  client.subscribe("$share/db-writers/connectmq/demo/s7h3/factory/+/telemetry",
    { qos: 1 }, (err, granted) => console.log(err || granted));
});
client.on("message", (topic, payload) => console.log(topic, payload.toString()));

How to Verify Shared Subscription Support

Support differs between brokers and between plans, so check rather than assume:

1. Read the SUBACK. In MQTT 5, a broker that does not support shared subscriptions returns reason code 0x9E (Shared Subscriptions not supported). The on_subscribe callback above prints it.
2. Check CONNACK. MQTT 5 brokers can advertise Shared Subscription Available = 0 in CONNACK. If the property is absent, support is assumed.
3. Run the split test. Two members, ten messages. If both members print all ten, the broker treated $share/... as an ordinary topic filter rather than a group.
4. Check the plan. Managed services often tie the feature to a tier. On ConnectMQ, shared subscriptions are included in the Serverless (Paid) plan and above.
5. Test with MQTT 3.1.1 separately. Some brokers accept $share from 3.1.1 clients as an extension; others require MQTT 5. Do not rely on it unless your broker documents it.

Shared Subscriptions Compared With Other Approaches

There are other ways to split work, and it helps to know when each fits. Manual topic partitioning (worker 1 subscribes to line1, worker 2 to line2) gives strict per-partition ordering but needs reconfiguration every time you add a worker or a line. A single consumer feeding an internal queue keeps ordering simple but makes that one process a bottleneck and a single point of failure. Shared subscriptions need no coordination between workers and scale by simply starting more instances, at the cost of per-topic ordering across workers. For most telemetry ingestion and alert processing, that trade-off is the right one.

Patterns and Pitfalls

• Group per purpose. Use one share name per independent job (db-writers, alerting, exporter). Each job then gets every message exactly once across its workers.
• Autoscaling. Adding a worker is simply connecting another client with the same filter; removing one is a disconnect. Give each instance a unique client ID, as explained in MQTT client ID best practices.
• Flow control. Combine shared subscriptions with a small MQTT 5 Receive Maximum on each worker so a slow instance is not handed a large burst of unacknowledged messages.
• Wildcards are fine. $share/group/factory/# spreads an entire branch. The matching rules are the usual ones, explained in MQTT wildcards.
• Do not use it for fan-out. If every consumer must see every message, use normal subscriptions.

New to MQTT in general? The MQTT tutorial covers subscriptions, QoS and sessions from the start.

Scale Consumers on ConnectMQ

ConnectMQ is a fully managed MQTT service supporting MQTT 3.1.1 and 5.0, with shared subscriptions available from the Serverless (Paid) plan, along with QoS 2, 1 MB messages and up to 1,000 connections. Dedicated and BYOC options are available when you need an isolated cluster. Sign up and start on the free Serverless plan with no credit card, then upgrade when your backend needs consumer groups. Connect your workers with YOUR_USERNAME, YOUR_PASSWORD and YOUR_CLIENT_ID.

Frequently Asked Questions

What is the syntax for an MQTT shared subscription?

Use $share/<ShareName>/<TopicFilter>, for example $share/workers/sensors/+/temperature. All clients using the same share name and filter form one group.

Do shared subscriptions work with MQTT 3.1.1?

They are defined in MQTT 5. Some brokers accept the $share syntax from 3.1.1 clients as an extension, but you should confirm this in your broker's documentation before relying on it.

Are retained messages delivered to shared subscriptions?

No. When a shared subscription is created, retained messages are not sent to it. Read retained state with a separate normal subscription if a worker needs it.

How does the broker choose which subscriber gets a message?

The MQTT specification leaves this to the broker. Round-robin and random selection are common, so never depend on a particular member receiving a particular message.