Shop Floor Data Collection: Methods, Tools and a Practical Architecture

Shop floor data collection is the foundation of every smart manufacturing project: no dashboard, OEE figure or predictive model is better than the data underneath it. This guide compares the four main collection methods, shows when each one fits, and explains how to centralize everything in one MQTT namespace so new machines and new consumers can be added without rework.

What Data Are You Actually Collecting?

Before picking a method, list the questions you want answered. Most plants start with a handful of signals that answer most daily questions:

State and counts are enough for OEE and downtime analysis, which is why they are the usual starting point. Reason codes come from people, not machines, so every architecture needs a way for operators to contribute. For designing those codes, see downtime reason codes.

Shop Floor Data Collection Methods Compared

1. Manual Collection

Paper sheets, spreadsheets or terminal entries filled in by operators. It costs almost nothing to start and captures context machines cannot know, such as why a line stopped. Its weaknesses are well known: entries are made at the end of a shift from memory, short stops disappear, and the data arrives too late to act on.

Manual collection remains useful for reason codes and checks, but it should be digital (a tablet form that publishes an event) and limited to what only a person can provide.

2. Reading the PLC

Modern machines have a PLC that already knows state, counts and many process values. Reading those tags over the machine's protocol (Modbus TCP, OPC UA, EtherNet/IP, S7 and others) gives the richest data with no new hardware.

The catch is access. Machine builders may not document tag addresses, the PLC may be locked, and changing PLC code can affect warranties. Coordinate with whoever owns the machine before writing anything to a controller; reading is usually low risk.

3. Retrofit Sensors

For older machines without accessible controllers, add sensors from the outside: a current transformer on the main motor to detect running versus idle, a proximity or photoelectric sensor to count parts, a vibration or temperature sensor for condition data, or a light sensor on the stack light. Wireless sensors make installation quick; see wireless industrial sensors for options.

Retrofit sensors see less than the PLC does, but they work on any machine regardless of age or brand, and they never touch the control system.

4. Edge Gateways

A gateway is a small industrial computer that talks to PLCs and sensors locally and forwards the data upstream. It handles protocol translation, buffering during network outages, filtering (publish on change instead of every scan) and adding context such as the machine's place in the plant hierarchy.

Most plants end up with a mix: gateways reading PLCs on newer lines, retrofit sensors on older ones and tablets for reason codes.

Centralizing Shop Floor Data with MQTT

The method you choose per machine matters less than where the data goes next. If every collector writes into its own database, you have recreated the silos you wanted to remove. The better pattern: every collector publishes to one MQTT broker using a consistent topic structure, and every consumer subscribes.

acme/plant1/machining/line1/cnc4/state      {"state":"RUNNING","ts":...}   retained
acme/plant1/machining/line1/cnc4/count      {"good":1,"scrap":0,"ts":...}
acme/plant1/machining/line1/cnc4/spindle    {"tempC":41.8,"rpm":8000,"ts":...}
acme/plant1/machining/line1/cnc4/reason     {"code":"TOOL_CHANGE","by":"op17"}

The path follows the ISA-95 hierarchy described in ISA-95 MQTT topics. Whether a value came from a PLC, a clamp-on sensor or a tablet becomes an implementation detail: consumers only see the topic.

A Minimal Gateway Example

The following Python script reads two holding registers from a Modbus TCP device (a state code and a part counter), publishes state on change as a retained message and publishes count increments. It uses pymodbus 3.x and paho-mqtt 2.x and runs on a Raspberry Pi or any small industrial PC.

import json
import time
import uuid

import paho.mqtt.client as mqtt
from pymodbus.client import ModbusTcpClient

PLC_IP = "192.168.10.50"
BASE = "connectmq/demo/sfdc-91be/line1/cnc4"   # replace 91be with your own id
STATES = {0: "STOPPED", 1: "RUNNING", 2: "IDLE", 3: "FAULT"}

mq = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2,
                 client_id=f"gw-cnc4-{uuid.uuid4().hex[:6]}")
mq.will_set(f"{BASE}/gateway", json.dumps({"online": False}), qos=1, retain=True)
mq.connect("broker.connectmq.com", 1883, keepalive=30)
mq.loop_start()
mq.publish(f"{BASE}/gateway", json.dumps({"online": True}), qos=1, retain=True)

plc = ModbusTcpClient(PLC_IP, port=502)
last_state, last_count = None, None

while True:
    if not plc.connected:
        plc.connect()
    rr = plc.read_holding_registers(0, count=2)
    if rr.isError():
        print("Modbus read failed:", rr)
        time.sleep(5)
        continue
    state = STATES.get(rr.registers[0], "UNKNOWN")
    count = rr.registers[1]
    now = time.time()

    if state != last_state:
        mq.publish(f"{BASE}/state", json.dumps({"state": state, "ts": now}), qos=1, retain=True)
        last_state = state

    if last_count is not None and count != last_count:
        delta = (count - last_count) % 65536   # handle 16-bit counter rollover
        mq.publish(f"{BASE}/count", json.dumps({"good": delta, "total": count, "ts": now}), qos=1)
    last_count = count
    time.sleep(1)

The same script works unchanged against a private ConnectMQ broker once you add credentials and TLS. The last will message marks the gateway offline if it loses its connection, so dashboards can distinguish "machine stopped" from "we have no data". On the public test broker everything is readable by anyone, so use it only with simulated values; for real machines switch to private credentials and TLS.

Shop Floor Data Collection Checklist

Use this list before rolling out to more than a pilot line:

1. Define the questions first and collect only the signals that answer them.
2. Agree on a topic hierarchy that mirrors your plant layout and document it.
3. Timestamp at the source (gateway or controller), in UTC.
4. Publish on change for states and counts; sample process values at a sensible fixed rate.
5. Retain current state so new subscribers see it immediately.
6. Mark data quality: gateway online/offline via last will, sensor faults as explicit values.
7. Handle counter rollover and gateway restarts without double counting.
8. Give each device its own client ID and credentials.
9. Let operators add context with a simple digital reason code form.
10. Check the data against reality for a week before anyone builds KPIs on it.

From Collection to Value

Once data flows, the next steps are straightforward: machine status monitoring on a screen at the line, OEE and downtime analysis, and later MES or maintenance integration. Each is just another subscriber.

The broker in the middle should be the least of your worries. ConnectMQ is a managed MQTT broker with MQTT 3.1.1 and 5.0, TLS and secure WebSocket endpoints, a live topic explorer and a devices view grouped by ISA-95 levels, which makes it easy to confirm that each collector publishes in the right place. Threshold alert rules in the portal can notify you when a value such as spindle temperature crosses a limit. The smart manufacturing guide shows how these pieces fit into a full plant roadmap, and the free public MQTT broker is the quickest way to test your gateway script before you create credentials.

Start Collecting Shop Floor Data with ConnectMQ

Create a broker on the free Serverless plan at app.connectmq.com, no credit card needed, and point your first gateway at it with the generated username, password and client ID.

Frequently Asked Questions

What is the cheapest way to start shop floor data collection?

A current sensor or part-count sensor on one bottleneck machine, a small gateway and a free broker. That alone gives you running time, idle time and output for the machine that limits your line.

Should I read data from the PLC or add sensors?

Read the PLC when you have access and documentation, because it already holds rich data. Use retrofit sensors on legacy or locked machines where PLC access is impossible or risky.

How often should machine data be collected?

States and counts should be published on change. Process values are usually sampled every 1 to 10 seconds, which is enough for dashboards and trend analysis without flooding the network.

Do I need an MES for shop floor data collection?

No. Collection, dashboards and OEE can run directly on MQTT data. An MES becomes useful later when you need order tracking, genealogy and enforced workflows.