What Is Edge AI? Running Models Next to the Sensors

What is edge AI? It is the practice of running machine learning inference on the device that collects the data, or on a gateway right beside it, instead of shipping every reading to a data centre. By the end of this explainer you will know how edge inference works, which hardware fits which job, and how to design the messages an edge device sends so the rest of your system gets answers rather than raw data.

Edge AI in One Paragraph

A trained model is a function: input data goes in, a prediction comes out. Training that function needs large datasets and lots of compute, so it usually happens in the cloud or on a workstation. Running it, called inference, is far cheaper. Edge AI takes the trained model, shrinks and optimizes it, and runs inference where the data is born: a camera, a vibration sensor, a microphone, a PLC gateway. The device then reports what it concluded, such as "person at gate", "bearing fault likely" or "part defective", over a lightweight protocol like MQTT.

How Edge Inference Works

The lifecycle looks the same whether the target is a microcontroller or a GPU module:

1. Collect and label data from the real sensors in the real environment.
2. Train a model with a framework such as TensorFlow, PyTorch or a tool like Edge Impulse.
3. Optimize it for the target. Typical steps are quantization (storing weights as 8-bit integers instead of 32-bit floats), pruning, and compiling for a specific accelerator.
4. Deploy it to the device as a file or firmware image.
5. Run inference in a loop: read the sensor, pre-process (resize an image, compute an FFT), run the model, post-process (apply thresholds, smooth over time).
6. Publish the result and occasionally upload samples for retraining.
7. Update the model when accuracy drifts, using a controlled rollout.

Step 6 is where edge AI meets IoT. Instead of streaming a video feed or thousands of accelerometer samples per second, the device publishes a few hundred bytes describing what happened.

Why Run AI at the Edge

There are trade-offs. Edge devices have limited memory and compute, so models must be small. Updating hundreds of devices is harder than updating one cloud service. And a model running in the field is harder to monitor than one running in a data centre. The edge AI vs cloud AI comparison goes deeper on when each approach wins and how hybrid designs combine them.

Edge AI Hardware Tiers

Edge AI spans a wide range of devices. A useful mental model is four tiers:

1. Microcontrollers (ESP32-S3, Arm Cortex-M boards). Kilobytes of RAM, milliwatts of power, models for keyword spotting, gesture recognition and vibration classification. This is the world of TinyML; see what TinyML is.
2. Single-board computers (Raspberry Pi class). Run small vision and audio models on the CPU at modest frame rates, and are easy to program in Python.
3. SBCs plus accelerators (Coral Edge TPU, Hailo modules). A dedicated neural processor offloads the model, making real-time object detection practical at low power.
4. GPU edge modules and industrial PCs (NVIDIA Jetson, PCs with GPUs). Multiple camera streams, larger models, segmentation and pose estimation.

Choose the lowest tier that meets your accuracy and latency needs: it is cheaper, uses less power and has fewer moving parts. The edge AI hardware comparison walks through specific boards.

Common Edge AI Use Cases

• Vision: person and vehicle detection, safety-zone monitoring, defect inspection, counting and occupancy.
• Audio: keyword spotting, abnormal machine sounds, glass-break detection.
• Vibration and current: bearing wear, imbalance and misalignment on motors and pumps.
• Environmental: occupancy inferred from CO2 and temperature patterns, leak detection.
• Industrial control: virtual sensors that estimate a value that is expensive to measure directly.

In each case, the device turns a dense signal into a sparse, meaningful event.

Publish Inference Results, Not Raw Data

The single most important design decision in edge AI for IoT is what goes on the wire. A good inference message is small, timestamped and self-describing:

{
  "ts": "2026-10-16T08:15:02Z",
  "device": "pump-07",
  "model": "vib-anomaly-v3",
  "label": "bearing_wear",
  "score": 0.87,
  "window_s": 2.0
}

Use a topic hierarchy that places the device in context, such as site/area/line/device/inference, and keep raw data local unless a subscriber genuinely needs it. ConnectMQ, for example, recognizes ISA-95 style topic hierarchies and groups devices by site, area and line in its devices view, which pays off once you have more than a handful of edge nodes. A cloud MQTT broker then fans the result out to dashboards, alerting, historians and cloud models without the edge device knowing who is listening.

Example: an Edge Inference Publisher in Python

The script below shows the pattern on any Linux edge device. The infer() function is a placeholder that you replace with TensorFlow Lite, ONNX Runtime or a vendor runtime. It publishes only when the label changes or every 60 seconds as a heartbeat, which keeps traffic low.

pip install "paho-mqtt>=2.0" numpy

import json
import time
import uuid
from datetime import datetime, timezone

import numpy as np
import paho.mqtt.client as mqtt

RUN = uuid.uuid4().hex[:8]
TOPIC = f"connectmq/demo/{RUN}/site1/pumps/pump-07/inference"
MODEL = "vib-anomaly-v3"


def read_window():
    """Placeholder sensor read: 2 s of vibration at 1 kHz."""
    t = np.arange(2000) / 1000
    signal = np.sin(2 * np.pi * 50 * t) + 0.3 * np.random.randn(t.size)
    if np.random.rand() < 0.1:
        signal += 0.8 * np.sin(2 * np.pi * 180 * t)   # simulated fault harmonic
    return signal


def infer(window):
    """Placeholder model: energy in a fault band. Replace with a real runtime."""
    spectrum = np.abs(np.fft.rfft(window))
    band = spectrum[170 * 2:190 * 2].sum() / spectrum.sum()
    score = float(min(1.0, band * 8))
    return ("bearing_wear" if score > 0.5 else "normal"), score


client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id=f"pump-07-{RUN}")
client.connect("broker.connectmq.com", 1883, keepalive=60)
client.loop_start()
print("publishing to", TOPIC)

last_label, last_sent = None, 0.0
while True:
    label, score = infer(read_window())
    now = time.time()
    if label != last_label or now - last_sent > 60:
        msg = {
            "ts": datetime.now(timezone.utc).isoformat(timespec="seconds"),
            "device": "pump-07",
            "model": MODEL,
            "label": label,
            "score": round(score, 3),
            "window_s": 2.0,
        }
        client.publish(TOPIC, json.dumps(msg), qos=1)
        print(msg)
        last_label, last_sent = label, now
    time.sleep(1)

Subscribe from another terminal with the printed topic to see the events:

mosquitto_sub -h broker.connectmq.com -p 1883 -i edge-viewer-5d1e -t 'connectmq/demo/<run-id>/#' -v

The free public MQTT broker is ideal for this kind of test, but anything published there is visible to anyone who subscribes to the topic. For real devices, use your own broker credentials and TLS on port 8883.

Edge AI Pitfalls to Plan For

1. Training data that does not match the field. Lab data rarely captures real lighting, noise or mounting. Collect from deployed sensors early.
2. No model version in messages. Without it, you cannot tell which model produced a bad alert.
3. Silent drift. Publish confidence scores and periodic statistics so you can see when the model becomes unsure.
4. Over-publishing. Sending every inference at full rate defeats the bandwidth advantage. Publish on change, on threshold or on a slow heartbeat.
5. No update path. Plan how a new model reaches devices before the first deployment; deploying ML models to edge devices covers the options.

For a broader view of where edge models sit alongside cloud analytics and LLM agents, read the AI and IoT guide.

Get Started with ConnectMQ

Edge AI devices need a dependable place to send their results. ConnectMQ is a managed MQTT broker supporting MQTT 3.1.1 and 5.0 over TCP, TLS and WebSockets, with a live topic explorer to watch inference events arrive. Create a broker on the free Serverless plan at app.connectmq.com, no credit card required, and point your edge devices at it.

Frequently Asked Questions

What is edge AI in simple terms?

Edge AI means a device runs a trained machine learning model itself, close to its sensors, instead of sending the data to a remote server for analysis. It then reports the result.

Is edge AI the same as TinyML?

TinyML is a subset of edge AI focused on microcontrollers with very little memory and power. Edge AI also includes larger devices such as Raspberry Pi boards, accelerators and GPU modules.

Does edge AI still need the cloud?

Usually yes, but for different jobs: training, fleet monitoring, long-term storage and combining results from many sites. The edge handles real-time inference and the cloud handles everything that benefits from a global view.

Why use MQTT for edge AI results?

Inference results are small, event-driven messages that many systems want to consume. MQTT's publish/subscribe model, QoS levels and low overhead fit that pattern well.