PubSubClient Buffer Size: Fixing Large Message Failures

If publish() returns false or incoming messages never reach your callback, the PubSubClient buffer size is the most likely cause. The library's default packet buffer is only 256 bytes, and anything larger fails without an error message. This guide explains exactly what the buffer must hold, how to size it with setBufferSize(), and how to stream large JSON payloads without allocating big buffers at all.

The Symptoms

Buffer problems are confusing because nothing crashes and the connection stays up. Watch for these signs:

• mqtt.publish() returns false for some messages but not others, typically the longer ones.
• Small test messages arrive in your callback, but the real JSON from your backend never does.
• Adding one more field to your JSON document suddenly stops telemetry.
• Retained configuration messages on a topic are visible in other clients but never reach the device.

All of these are explained by one rule: the entire MQTT packet, not just the payload, must fit into PubSubClient's internal buffer.

What the Buffer Must Hold

PubSubClient uses a single buffer for both outgoing and incoming packets. For a PUBLISH packet, that buffer holds:

The library reserves a few bytes of header space when it builds the packet, so in practice you need topic length + payload length + about 7 bytes, plus 2 more for incoming QoS 1 messages. With the 256-byte default and a topic such as connectmq/demo/sensor-4d2a/telemetry (36 characters), your payload must stay under roughly 210 bytes. A JSON document with ten readings and timestamps passes that easily.

The same limit applies to other packets the library builds in the buffer, including CONNECT (client ID, credentials and will message) and SUBSCRIBE. A very long password plus a long will message can push even the CONNECT packet over 256 bytes, which shows up as a failed connect().

Fix 1: Increase the Buffer with setBufferSize()

Since PubSubClient 2.8 you can resize the buffer at runtime. Older tutorials tell you to edit MQTT_MAX_PACKET_SIZE in PubSubClient.h; that still works but is fragile, because library updates overwrite it and the change silently disappears.

#include <WiFi.h>
#include <PubSubClient.h>

WiFiClient net;
PubSubClient mqtt(net);

void setup() {
  Serial.begin(115200);
  // ... connect Wi-Fi ...

  mqtt.setServer("broker.connectmq.com", 1883);
  if (!mqtt.setBufferSize(2048)) {
    Serial.println("Could not allocate MQTT buffer");
  }
  Serial.printf("MQTT buffer size: %u\n", mqtt.getBufferSize());
}

Key points:

1. setBufferSize() takes a uint16_t, so the theoretical ceiling is 65,535 bytes.
2. It reallocates memory and returns false if allocation fails, leaving the previous buffer in place. Always check the return value.
3. Call it before connect(). Resizing while connected works but is wasteful.
4. getBufferSize() confirms the current size.

How large should it be? Use the formula above for your largest expected message, then add a margin of 10 to 20 percent. Do not set 16 KB "just in case" on an ESP8266: that RAM is then unavailable for TLS, and the next handshake may fail. On an ESP32 a few kilobytes is no problem.

Fix 2: Stream Large Payloads with beginPublish()

Increasing the buffer is fine for occasional large messages, but streaming avoids the RAM cost entirely. beginPublish() writes the header and topic, then lets you write the payload directly to the network client in pieces. Only the header goes through the buffer, so a 256-byte buffer can send a payload of many kilobytes.

The catch: you must know the payload length in advance, because MQTT puts it in the header. ArduinoJson makes this easy with measureJson().

#include <WiFi.h>
#include <PubSubClient.h>
#include <ArduinoJson.h>

const char* TOPIC = "connectmq/demo/sensor-4d2a/batch";

WiFiClient net;
PubSubClient mqtt(net);

bool publishBatch(const float* values, size_t count) {
  JsonDocument doc;                 // ArduinoJson 7
  doc["device"] = "sensor-4d2a";
  doc["uptime"] = millis() / 1000;
  JsonArray arr = doc["values"].to<JsonArray>();
  for (size_t i = 0; i < count; i++) arr.add(values[i]);

  size_t len = measureJson(doc);
  if (!mqtt.beginPublish(TOPIC, len, false)) {
    Serial.println("beginPublish failed");
    return false;
  }
  serializeJson(doc, mqtt);         // PubSubClient is a Print, so this streams
  return mqtt.endPublish() == 1;
}

With 200 float values the payload is far larger than 256 bytes, yet the default buffer is enough. beginPublish(), write()/print() and endPublish() are available in PubSubClient 2.7 and later.

Note that the broker still sees one MQTT message, and it must be within the broker's limit. On the public MQTT test broker at broker.connectmq.com, the limit is 64 KB per message.

Receiving Large Messages

Streaming only helps outgoing messages. For incoming messages PubSubClient reads the whole packet into the buffer before calling your callback. If the packet is larger than the buffer, the library reads and discards it, and your callback is never called. There is no error and no log.

So for devices that receive large configuration documents or firmware manifests:

• Set the buffer to the largest incoming message plus topic plus overhead.
• Keep incoming messages compact. Send only the fields the device needs rather than a full backend object.
• Split big data across several topics, for example config/network and config/sensors, each well under the limit.
• For really large payloads such as firmware images, publish a URL over MQTT and download the file over HTTPS.

Test what happens with a message that is deliberately too large:

mosquitto_pub -h broker.connectmq.com -p 1883 -i sizetest-4d2a \
  -t "connectmq/demo/sensor-4d2a/cmd" \
  -m "$(head -c 400 /dev/zero | tr '\0' 'x')"

With the default buffer, the device ignores it. After setBufferSize(1024), the callback fires with a length of 400.

Debugging Checklist

Work through this list when messages go missing:

1. Print mqtt.getBufferSize() after setup to confirm the size you think you set.
2. Log the topic and payload length before each publish: Serial.printf("%u + %u\n", strlen(topic), strlen(payload));.
3. Check publish()'s return value and print mqtt.connected() when it fails, to separate size problems from connection problems.
4. On ESP8266, watch ESP.getMaxFreeBlockSize(). A large buffer plus TLS can exhaust contiguous memory.
5. Compare with a desktop client: if mosquitto_sub receives the message but the device does not, the buffer is too small.
6. Make sure your message is within your broker's limit as well as the device buffer.

If the connection itself fails, the cause is elsewhere; see PubSubClient rc=-2 and other state codes. For the complete library walkthrough, read our PubSubClient tutorial. If you need a library that streams incoming messages too, the ArduinoMqttClient guide shows an alternative approach, and the ESP32 IoT tutorial ties these techniques into a full project.

Broker Message Limits

The device buffer is only half the story. Every broker caps message size too, and messages above the limit are rejected or cause a disconnect:

For embedded devices, staying in the low kilobytes is good practice anyway. Smaller messages mean less RAM, faster TLS records and fewer retransmissions on weak Wi-Fi.

Send Larger Payloads with ConnectMQ

If your telemetry has outgrown the public broker, a managed MQTT broker gives you private credentials, TLS and higher message limits. ConnectMQ is a cloud MQTT broker with MQTT 3.1.1 and 5.0, an in-browser MQTT client and a live topic explorer where you can inspect exactly what your devices send. Create a free Serverless broker at app.connectmq.com, no credit card required.

Frequently Asked Questions

What is the default PubSubClient buffer size?

256 bytes, defined as MQTT_MAX_PACKET_SIZE. It covers the whole packet, including the topic and headers, so the usable payload is smaller.

Should I edit MQTT_MAX_PACKET_SIZE or use setBufferSize()?

Use setBufferSize() in PubSubClient 2.8 or later. It survives library updates and lets you check whether the allocation succeeded.

Why does publish() return false with a large message?

The topic plus payload plus header exceeds the buffer. Increase the buffer or use beginPublish() to stream the payload.

Can PubSubClient receive messages larger than its buffer?

No. Oversized incoming messages are discarded before your callback runs. Increase the buffer or keep incoming messages smaller.