ESP32-CAM Person Detection with On-Device AI and MQTT Alerts

ESP32-CAM person detection puts a small neural network on a board that costs about as much as a sandwich, and turns it into a presence sensor that publishes MQTT alerts. In this tutorial you will run the TensorFlow Lite Micro person detection model on an AI-Thinker ESP32-CAM, publish a confidence score and debounced alerts to broker.connectmq.com, and learn the memory and timing pitfalls that trip up most first attempts.

What the Board Can and Cannot Do

The ESP32-CAM pairs a dual-core ESP32 with an OV2640 camera and, on most boards, 4 MB of PSRAM. That is enough for a carefully designed TinyML model, but not for a general object detector.

The model used here, from the TensorFlow Lite Micro examples, answers one question per frame: is there a person in this picture? It does not tell you where or how many. For that, publish an event to a bigger model, or see YOLO object detection over MQTT.

Parts and Software

• AI-Thinker ESP32-CAM with PSRAM, plus a USB-to-serial adapter or ESP32-CAM-MB programmer board
• A stable 5 V supply capable of at least 500 mA (brownouts are the number one problem)
• Arduino IDE 2 with the esp32 board package by Espressif, board "AI Thinker ESP32-CAM"
• Libraries: PubSubClient and TensorFlowLite_ESP32 (Library Manager)
• From the TensorFlowLite_ESP32 person_detection example: copy person_detect_model_data.h and person_detect_model_data.cpp into your sketch folder

If you have never connected an ESP32 to MQTT before, the ESP32 MQTT tutorial covers the basics, and ESP32-CAM with MQTT covers publishing JPEG snapshots.

How the Pipeline Works

1. The camera captures a 96 x 96 grayscale frame directly, so no resizing is needed.
2. Pixels are converted from unsigned 0 to 255 to the signed int8 range the quantized model expects.
3. The interpreter runs the model and returns two scores: "no person" and "person".
4. The sketch publishes the person score periodically and raises an alert only when the score stays above a threshold for several frames.
5. A cooldown prevents a single person standing in view from generating an alert every second.

The Complete Sketch

Replace the Wi-Fi credentials and the topic ID. The public area of broker.connectmq.com accepts anonymous connections with any non-empty client ID, but everything published there is readable by others, so publish scores and alerts only, never images of real people.

#include <WiFi.h>
#include <PubSubClient.h>
#include "esp_camera.h"
#include "esp_heap_caps.h"

#include <TensorFlowLite_ESP32.h>
#include "tensorflow/lite/micro/micro_error_reporter.h"
#include "tensorflow/lite/micro/micro_interpreter.h"
#include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
#include "tensorflow/lite/schema/schema_generated.h"
#include "person_detect_model_data.h"

const char* WIFI_SSID = "YOUR_WIFI";
const char* WIFI_PASS = "YOUR_WIFI_PASSWORD";
const char* MQTT_HOST = "broker.connectmq.com";
const uint16_t MQTT_PORT = 1883;
const char* PREFIX = "connectmq/demo/pd-e51a/home/frontdoor/cam1";

const int PERSON_THRESHOLD = 60;   // percent
const int FRAMES_TO_RAISE = 3;
const unsigned long COOLDOWN_MS = 30000;
const unsigned long SCORE_EVERY_MS = 5000;

// AI-Thinker ESP32-CAM pin map
#define PWDN_GPIO_NUM 32
#define RESET_GPIO_NUM -1
#define XCLK_GPIO_NUM 0
#define SIOD_GPIO_NUM 26
#define SIOC_GPIO_NUM 27
#define Y9_GPIO_NUM 35
#define Y8_GPIO_NUM 34
#define Y7_GPIO_NUM 39
#define Y6_GPIO_NUM 36
#define Y5_GPIO_NUM 21
#define Y4_GPIO_NUM 19
#define Y3_GPIO_NUM 18
#define Y2_GPIO_NUM 5
#define VSYNC_GPIO_NUM 25
#define HREF_GPIO_NUM 23
#define PCLK_GPIO_NUM 22

WiFiClient net;
PubSubClient mqtt(net);
char clientId[32];

constexpr int kArenaSize = 100 * 1024;
uint8_t* tensorArena = nullptr;
tflite::MicroErrorReporter errorReporter;
tflite::MicroInterpreter* interpreter = nullptr;
TfLiteTensor* input = nullptr;

int hits = 0;
unsigned long lastAlert = 0, lastScore = 0;

bool initCamera() {
  camera_config_t c = {};
  c.ledc_channel = LEDC_CHANNEL_0;
  c.ledc_timer = LEDC_TIMER_0;
  c.pin_d0 = Y2_GPIO_NUM; c.pin_d1 = Y3_GPIO_NUM; c.pin_d2 = Y4_GPIO_NUM;
  c.pin_d3 = Y5_GPIO_NUM; c.pin_d4 = Y6_GPIO_NUM; c.pin_d5 = Y7_GPIO_NUM;
  c.pin_d6 = Y8_GPIO_NUM; c.pin_d7 = Y9_GPIO_NUM;
  c.pin_xclk = XCLK_GPIO_NUM; c.pin_pclk = PCLK_GPIO_NUM;
  c.pin_vsync = VSYNC_GPIO_NUM; c.pin_href = HREF_GPIO_NUM;
  c.pin_sccb_sda = SIOD_GPIO_NUM; c.pin_sccb_scl = SIOC_GPIO_NUM;
  c.pin_pwdn = PWDN_GPIO_NUM; c.pin_reset = RESET_GPIO_NUM;
  c.xclk_freq_hz = 20000000;
  c.pixel_format = PIXFORMAT_GRAYSCALE;
  c.frame_size = FRAMESIZE_96X96;
  c.fb_count = 1;
  c.fb_location = CAMERA_FB_IN_PSRAM;
  c.grab_mode = CAMERA_GRAB_LATEST;
  return esp_camera_init(&c) == ESP_OK;
}

bool initModel() {
  tensorArena = (uint8_t*)heap_caps_malloc(kArenaSize, MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT);
  if (!tensorArena) return false;
  const tflite::Model* model = tflite::GetModel(g_person_detect_model_data);
  static tflite::MicroMutableOpResolver<5> resolver;
  resolver.AddAveragePool2D();
  resolver.AddConv2D();
  resolver.AddDepthwiseConv2D();
  resolver.AddReshape();
  resolver.AddSoftmax();
  static tflite::MicroInterpreter staticInterpreter(
      model, resolver, tensorArena, kArenaSize, &errorReporter);
  interpreter = &staticInterpreter;
  if (interpreter->AllocateTensors() != kTfLiteOk) return false;
  input = interpreter->input(0);
  return true;
}

void connectMqtt() {
  while (!mqtt.connected()) {
    String status = String(PREFIX) + "/status";
    if (mqtt.connect(clientId, status.c_str(), 1, true, "offline")) {
      mqtt.publish(status.c_str(), "online", true);
    } else {
      Serial.printf("MQTT failed, state=%d\n", mqtt.state());
      delay(3000);
    }
  }
}

void setup() {
  Serial.begin(115200);
  if (!initCamera()) { Serial.println("Camera init failed"); ESP.restart(); }
  if (!initModel()) { Serial.println("Model init failed, increase arena"); ESP.restart(); }

  WiFi.mode(WIFI_STA);
  WiFi.begin(WIFI_SSID, WIFI_PASS);
  while (WiFi.status() != WL_CONNECTED) delay(300);
  snprintf(clientId, sizeof(clientId), "esp32cam-%06lx", (unsigned long)(ESP.getEfuseMac() & 0xFFFFFF));
  mqtt.setServer(MQTT_HOST, MQTT_PORT);
  mqtt.setKeepAlive(30);
  connectMqtt();
}

void loop() {
  if (!mqtt.connected()) connectMqtt();
  mqtt.loop();

  camera_fb_t* fb = esp_camera_fb_get();
  if (!fb) return;
  for (size_t i = 0; i < input->bytes && i < fb->len; i++) {
    input->data.int8[i] = (int8_t)(fb->buf[i] - 128);
  }
  esp_camera_fb_return(fb);

  unsigned long t0 = millis();
  if (interpreter->Invoke() != kTfLiteOk) return;
  unsigned long inferMs = millis() - t0;

  TfLiteTensor* out = interpreter->output(0);
  int personPct = (out->data.int8[1] + 128) * 100 / 255;

  hits = personPct >= PERSON_THRESHOLD ? hits + 1 : 0;
  unsigned long now = millis();

  if (hits == FRAMES_TO_RAISE && now - lastAlert > COOLDOWN_MS) {
    lastAlert = now;
    char msg[96];
    snprintf(msg, sizeof(msg), "{\"event\":\"person\",\"score\":%d,\"uptimeMs\":%lu}", personPct, now);
    mqtt.publish((String(PREFIX) + "/alert").c_str(), msg);
    Serial.println(msg);
  }
  if (now - lastScore > SCORE_EVERY_MS) {
    lastScore = now;
    char msg[96];
    snprintf(msg, sizeof(msg), "{\"person\":%d,\"inferMs\":%lu,\"rssi\":%d}", personPct, inferMs, WiFi.RSSI());
    mqtt.publish((String(PREFIX) + "/score").c_str(), msg);
  }
}

Open the Serial Monitor at 115200 baud and subscribe from your computer:

mosquitto_sub -h broker.connectmq.com -p 1883 -i pd-watch-e51a -t "connectmq/demo/pd-e51a/#" -v

Step in front of the camera and an alert should appear after a few frames; the score topic updates every five seconds.

Tuning ESP32-CAM Person Detection

• Threshold. Start at 60 percent. Lower it if people at the edge of the frame are missed, raise it if shadows or posters trigger alerts.
• Frames to raise. Three consecutive frames filters most single-frame glitches at the cost of a slightly later alert.
• Cooldown. Set it to how often you actually want to be notified. A doorway might need 30 seconds; a restricted area might need every entry.
• Lighting. The model was trained on varied images but struggles in very low light. An infrared-capable camera module with IR illumination helps at night.
• Field of view. The model sees only 96 x 96 pixels, so a person must fill a reasonable portion of the frame. Mount the camera so that the area of interest is close.

Power and Placement

Running a camera and a neural network continuously keeps the ESP32 busy and Wi-Fi active, so this design suits mains-powered locations. For battery operation, pair the board with a passive infrared (PIR) sensor: let the ESP32 sleep, wake it when the PIR triggers, run person detection on a handful of frames to confirm that the motion was a person rather than a pet or a curtain, publish the result and go back to sleep. The MQTT payload stays the same, so subscribers do not care which mode the camera runs in.

Placement matters as much as the model. Mount the camera at a height and angle where people appear upright and reasonably large in the frame, avoid pointing it at windows that cause strong backlight, and keep it out of reach of rain unless the enclosure is rated for it.

Building on the Alerts

Once alerts arrive on a topic, anything that speaks MQTT can react: Home Assistant can turn on a light, Node-RED can send a phone notification, and a Raspberry Pi running a larger detector can request a snapshot and count people properly. With ConnectMQ in the middle, each of these is simply another subscriber, and you can add or remove them without reflashing the camera.

Troubleshooting

1. "Brownout detector was triggered" and resets. Use a better 5 V supply and a short, thick cable; the camera and Wi-Fi together draw current spikes.
2. Camera init failed. Check that the board selection is AI Thinker ESP32-CAM and that PSRAM is enabled; also verify the ribbon cable is seated.
3. Model init failed. The tensor arena is too small or internal RAM is fragmented; increase kArenaSize in steps of 10 KB and initialize the model before Wi-Fi.
4. Compile errors in TensorFlow headers. Library versions differ in the MicroInterpreter constructor; match the include paths and constructor to the example shipped with your installed library version.
5. MQTT state -2 or constant reconnects. Wi-Fi is weak or the client ID is duplicated. Move the board closer to the access point and make sure no other device uses the same client ID.

Securing It for Real Use

A camera that watches a real doorway produces personal data, even if only a score leaves the device. For anything beyond a test, move to a private broker with TLS: use WiFiClientSecure instead of WiFiClient, connect to port 8883 and authenticate with mqtt.connect("YOUR_CLIENT_ID", "YOUR_USERNAME", "YOUR_PASSWORD", ...). The public MQTT broker for testing is ideal for development, while a fully managed MQTT service such as ConnectMQ gives each camera its own credentials and keeps traffic encrypted.

Because the model is a person classifier, a natural extension is to use it as a trigger: when it fires, publish a snapshot to a separate topic or wake a more capable model elsewhere. For more on squeezing models onto microcontrollers, read TinyML on the ESP32 and the broader AI and IoT guide.

Get Started with ConnectMQ

Create a private broker on the free Serverless plan at app.connectmq.com, no credit card required, then switch the sketch to TLS and the generated credentials. You can watch alerts arrive in the portal's live topic explorer and add a threshold alert rule on the person score.

Frequently Asked Questions

How fast is person detection on an ESP32-CAM?

Expect a few inferences per second at most, depending on library version and whether optimized kernels are enabled. That is plenty for presence detection but not for tracking motion.

Can the ESP32-CAM count people or draw bounding boxes?

Not with this model, which only classifies a whole frame as person or no person. For counting and boxes, send events to a Raspberry Pi or Jetson running a detector.

Do I need PSRAM for person detection?

The camera frame buffer is placed in PSRAM in this sketch, and most ESP32-CAM boards include it. The tensor arena itself is allocated in faster internal RAM.

Is it safe to send images to the public broker?

No. The public area is readable by anyone, so publish only scores and alerts there and keep any images on a private broker with TLS and credentials.