Sensor data outlier detection does not need machine learning to be useful. Three classic techniques, the Z-score, the interquartile range (IQR) and the exponentially weighted moving average (EWMA), catch most spikes, dropouts and drifts in temperature streams. This article explains how each works, where it breaks, and gives you Python code that applies all three to a live MQTT feed.
Outliers vs Anomalies in Sensor Streams
An outlier is a reading that sits far from the others. An anomaly is a reading or pattern that indicates something unusual is happening in the real world. The two overlap but are not identical. A single wild reading from electrical noise is an outlier and not a process anomaly; a slow two-degree rise over an hour may never produce an outlier yet signal a failing cooling fan.
Sensor data outlier detection therefore has two jobs:
- Cleaning. Remove or flag spikes and impossible values before they reach dashboards, averages and models.
- Early warning. Highlight readings that deserve attention because they break the recent pattern.
The methods below handle both, with different strengths. For the broader architecture, including machine learning detectors, see IoT anomaly detection methods.
Z-Score: Distance in Standard Deviations
The Z-score measures how many standard deviations a value is from the mean:
z = (x - mean) / std
A reading with an absolute Z-score above 3 is a common outlier rule. In a streaming context you compute the mean and standard deviation over a rolling window of recent readings rather than all history, so the baseline follows slow changes.
Strengths: easy to explain, fast, works well when the signal is roughly normal around a stable level.
Weaknesses: the mean and standard deviation are themselves distorted by outliers. A burst of spikes inflates the standard deviation and hides the next spike. It also assumes symmetric noise, which many sensors do not have.
A robust variant replaces mean with median and standard deviation with the median absolute deviation (MAD). That version is used in the real-time anomaly detection in Python tutorial.
IQR: Quartile Fences
The interquartile range uses percentiles instead of averages:
Q1 = 25th percentile, Q3 = 75th percentile, IQR = Q3 - Q1
lower fence = Q1 - 1.5 * IQR
upper fence = Q3 + 1.5 * IQR
Anything outside the fences is an outlier. Use 3 instead of 1.5 for "extreme" outliers.
Strengths: robust to the outliers it is looking for, no normality assumption, handles skewed data such as vibration amplitudes or current draw.
Weaknesses: needs a window large enough for stable quartiles, and on heavily quantized sensors (a temperature sensor with 0.5 degree resolution, for example) the IQR can collapse to zero and flag every small change. Add a minimum fence width equal to the sensor's resolution.
EWMA: Smoothed Baseline with Control Limits
EWMA keeps a running smoothed value where recent readings weigh more:
ewma = alpha * x + (1 - alpha) * ewma_previous
Alongside it you track an exponentially weighted variance, and flag readings that fall outside ewma ± k * ewm_std. A small alpha (0.01 to 0.05) gives a slow, stable baseline; a larger alpha follows changes quickly.
Strengths: constant memory (two numbers per signal), ideal for microcontrollers, naturally adapts to slow drift, and sensitive to small sustained shifts when used as a control chart.
Weaknesses: because it adapts, a slow fault can be absorbed into the baseline. Pair it with an absolute limit for values that must never be exceeded.
Comparing the Methods
| Method | Memory per signal | Robust to outliers | Handles skew | Detects slow drift | Fits on microcontroller |
|---|---|---|---|---|---|
| Rolling Z-score | Window of N values | No | No | Poorly | Yes, small N |
| Robust Z (median/MAD) | Window of N values | Yes | Partly | Poorly | Yes, small N |
| IQR fences | Window of N values | Yes | Yes | Poorly | Yes, small N |
| EWMA control limits | Two numbers | Partly | No | Yes, with small alpha | Easily |
No single method wins. A practical combination for temperature is: hard physical limits, IQR for spikes, and EWMA with a small alpha for drift.
Sensor Data Outlier Detection in Python
The code below implements all three detectors as small classes with an update method, so you can run them side by side on the same stream.
# detectors.py
import math
import statistics
from collections import deque
class RollingZ:
def __init__(self, window=60, k=3.0):
self.buf, self.k = deque(maxlen=window), k
def update(self, x):
flagged = False
if len(self.buf) >= 20:
mean = statistics.fmean(self.buf)
std = statistics.pstdev(self.buf) or 1e-9
flagged = abs(x - mean) / std > self.k
if not flagged:
self.buf.append(x)
return flagged
class IQRFence:
def __init__(self, window=60, k=1.5, min_width=0.2):
self.buf, self.k, self.min_width = deque(maxlen=window), k, min_width
def update(self, x):
flagged = False
if len(self.buf) >= 20:
q1, _, q3 = statistics.quantiles(self.buf, n=4)
iqr = max(q3 - q1, self.min_width)
flagged = x < q1 - self.k * iqr or x > q3 + self.k * iqr
if not flagged:
self.buf.append(x)
return flagged
class EWMA:
def __init__(self, alpha=0.05, k=4.0, warmup=20):
self.alpha, self.k, self.warmup = alpha, k, warmup
self.mean, self.var, self.n = None, 0.0, 0
def update(self, x):
self.n += 1
if self.mean is None:
self.mean = x
return False
diff = x - self.mean
std = math.sqrt(self.var) or 1e-9
flagged = self.n > self.warmup and abs(diff) > self.k * std
if not flagged:
self.mean += self.alpha * diff
self.var = (1 - self.alpha) * (self.var + self.alpha * diff * diff)
return flagged
Each detector skips learning from values it flags, which keeps a spike from contaminating the baseline.
Applying It to a Live MQTT Temperature Stream
To try this end to end, publish simulated temperatures to the public MQTT test broker at broker.connectmq.com and run the detectors in a subscriber. Replace t9x4q2 with your own random ID; the public area is readable by everyone, so only use synthetic data.
# temp_sim.py
import json, random, time, uuid
import paho.mqtt.client as mqtt
TOPIC = "connectmq/demo/t9x4q2/lab/room1/temperature"
c = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id=f"tsim-{uuid.uuid4().hex[:6]}")
c.connect("broker.connectmq.com", 1883, keepalive=60)
c.loop_start()
base = 21.0
while True:
base += random.gauss(0, 0.01) # slow wander
value = base + random.gauss(0, 0.15)
if random.random() < 0.02:
value += random.choice([-6, 7]) # occasional spike
c.publish(TOPIC, json.dumps({"tempC": round(value, 2), "ts": time.time()}))
time.sleep(1)
# temp_outliers.py
import json, uuid
import paho.mqtt.client as mqtt
from detectors import RollingZ, IQRFence, EWMA
TOPIC = "connectmq/demo/t9x4q2/lab/room1/temperature"
detectors = {"z": RollingZ(), "iqr": IQRFence(), "ewma": EWMA()}
def on_connect(client, userdata, flags, reason_code, properties):
client.subscribe(TOPIC)
def on_message(client, userdata, msg):
x = json.loads(msg.payload)["tempC"]
flags = {name: d.update(x) for name, d in detectors.items()}
if any(flags.values()):
print(f"{x:6.2f} flagged by: {[n for n, f in flags.items() if f]}")
client.publish(TOPIC.rsplit("/", 1)[0] + "/outliers",
json.dumps({"tempC": x, "flags": flags}), qos=1)
c = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id=f"tout-{uuid.uuid4().hex[:6]}")
c.on_connect = on_connect
c.on_message = on_message
c.connect("broker.connectmq.com", 1883, keepalive=60)
c.loop_forever()
Run both and watch which detectors agree. You will typically see all three catch the large spikes, while the Z-score occasionally misses a spike that follows closely after another one, which is exactly the weakness described above.
Detecting on the Device vs After the Broker
You can run these detectors in two places, and the choice affects what you publish.
Running on the sensor node, for example an ESP32, means the device can attach an outlier flag to each message or publish only when something interesting happens. That saves bandwidth and works even when the network is down, but the thresholds are baked into firmware and harder to change.
Running after the broker, in a subscriber like the one above, keeps devices simple and lets you change methods and thresholds at any time, compare several detectors in parallel and reprocess history. The cost is that every raw reading must travel to the broker.
A common compromise is a hard limit and stuck-value check on the device, which catches broken sensors immediately, and the statistical detectors in a subscriber service where they are easy to tune. With a ConnectMQ broker in the middle, both outputs land on topics that the same dashboards can read.
Practical Checklist
- Apply hard physical limits first (a room sensor reading 150 degrees is a fault, not an outlier).
- Detect stuck values separately by checking for identical readings over a period.
- Choose window sizes in time, not samples, if publishing intervals vary.
- Set minimum band widths to the sensor's resolution.
- Flag outliers rather than deleting them, and keep the raw value for later analysis.
- Require two or three consecutive flags before alerting a person.
- Revisit thresholds after a week of real data.
When signals interact, for example current and flow on a pump, these per-signal methods reach their limit and a multivariate model such as Isolation Forest on sensor data becomes worthwhile.
Where the Broker Fits
Every piece above is an ordinary MQTT client: the sensor publishes, the detector subscribes and republishes. The MQTT broker simply routes messages, which means you can add, remove or replace detectors without touching devices. For real deployments, a private MQTT broker in the cloud such as ConnectMQ adds TLS, credentials and a live topic explorer where you can watch raw readings and flagged outliers side by side. For more ways to combine analytics with devices, see the AI and IoT guide.
Get Started with ConnectMQ
Create a broker on the free Serverless plan at app.connectmq.com with no credit card, point your sensors at it and run the outlier detector against your own data instead of the simulator.
Frequently Asked Questions
Which outlier detection method is best for temperature sensors?
A combination works best: physical limits, IQR fences for spikes and EWMA with a small smoothing factor for slow drift. Temperature changes slowly, so EWMA is particularly effective.
What window size should I use for rolling statistics?
Cover enough time to capture normal noise but not so much that legitimate changes take ages to absorb. For a reading every second, one to five minutes is a reasonable starting point.
Should outliers be removed from the data?
Flag them instead of deleting them. Keep the raw value with an outlier flag so you can audit decisions and retrain models later.
Can these methods run on an ESP32?
Yes. EWMA needs only two floats per signal, and a rolling window of 30 to 60 values fits easily in ESP32 memory, so you can flag outliers before publishing.
