Unified Namespace vs Data Lake: Complementary or Competing?

The unified namespace vs data lake debate usually starts when an IT team already has a data lake and an OT team proposes a UNS, and both wonder whether one makes the other redundant. In this article you will learn what each architecture is actually for, where they overlap, why they work best together, and how to move data from an MQTT-based namespace into lake storage with a small, reliable subscriber.

Two Different Questions

The simplest way to separate the two is by the question each answers.

A unified namespace answers "what is happening right now?" It is an event-driven hub, normally an MQTT broker, where every machine and system publishes its current state under a structured topic tree. Consumers subscribe and react in milliseconds to seconds. The namespace keeps the latest value of each topic, typically as a retained message, and not much more.

A data lake answers "what happened, and what does it mean?" It is a storage layer, usually object storage with files in open formats such as Parquet, plus query engines on top. It keeps months or years of raw and refined data so analysts and data scientists can join production data with quality, maintenance, ERP and energy data.

One is a nervous system; the other is long-term memory. Asking which one to choose is like asking whether a plant needs its SCADA or its reporting database.

Unified Namespace vs Data Lake at a Glance

Read down any row and the conclusion is the same: these are layers, not alternatives.

Where the Confusion Comes From

There are three understandable reasons people treat them as competitors.

First, both promise to break down data silos. A UNS does it by giving every system one place to publish and subscribe; a data lake does it by putting every system's data into one storage layer. Same goal, different time scale.

Second, some vendors blur the line. Brokers with built-in persistence and lakes with streaming ingestion both reach towards the other side. Those features exist, but stretching either tool too far creates problems: brokers are not designed to answer "average temperature per shift last quarter", and lakes are not designed to push a stop signal to a line in under a second.

Third, early UNS projects sometimes skip storage entirely, then discover that operations want trends and engineers want root-cause analysis. The fix is not to abandon the UNS; it is to add a storage consumer.

How They Fit Together

The pattern that works is straightforward: the UNS is the single source for live data, and the data lake is one of its subscribers.

PLCs / gateways / Sparkplug edge nodes
              |
              v
   Unified namespace (MQTT broker, ISA-95 topics)
     |            |              |               |
     v            v              v               v
 Dashboards    Alerting     Historian      Lake ingestion
                            (hot data)     (raw files, then
                                            curated tables)

Benefits of this arrangement:

• One integration per source. A PLC gateway publishes once; the lake does not need its own connector to every machine.
• Context comes free. Because topics follow the ISA-95 hierarchy, the ingestion job can turn site, area, line and machine into columns without a lookup table.
• Independent evolution. You can replace the lake platform without touching the plant, and add new plant data without changing the lake pipeline.
• Hot and cold paths. A historian holds recent high-resolution data for operators; the lake holds the long tail for analytics. Our MQTT historian guide compares historian options.

If you are starting from zero, build the namespace first. Our how to build a unified namespace tutorial walks through it, and the lake ingestion below plugs straight into it.

Moving Data From the UNS Into a Data Lake

Ingestion is just another MQTT client. The script below subscribes to a namespace branch and writes every message as a line of JSON into files partitioned by date and hour, a common landing-zone layout for a lake. A scheduled job can later convert the files to Parquet and load curated tables.

It assumes topics shaped like <prefix>/<site>/<area>/<line>/<machine>/<kind> and JSON payloads. Install the client with pip install "paho-mqtt>=2.0".

import json
import os
from datetime import datetime, timezone

import paho.mqtt.client as mqtt

BROKER = os.getenv("MQTT_HOST", "broker.connectmq.com")
PORT = int(os.getenv("MQTT_PORT", "1883"))
PREFIX = os.getenv("UNS_PREFIX", "connectmq/demo/k4p9")
LANDING = os.getenv("LANDING_DIR", "./landing/uns")

def on_connect(client, userdata, flags, reason_code, properties):
    print("ingestion connected:", reason_code)
    client.subscribe(f"{PREFIX}/+/+/+/+/telemetry", qos=1)

def on_message(client, userdata, msg):
    if msg.retain:
        return  # skip retained replay so restarts do not duplicate rows
    levels = msg.topic[len(PREFIX) + 1:].split("/")
    if len(levels) != 5:
        return
    site, area, line, machine, _ = levels
    try:
        data = json.loads(msg.payload)
    except ValueError:
        print("skipping non-JSON payload on", msg.topic)
        return
    received = datetime.now(timezone.utc)
    row = {
        "received_at": received.isoformat(),
        "site": site, "area": area, "line": line, "machine": machine,
        **data,
    }
    folder = os.path.join(LANDING, f"date={received:%Y-%m-%d}", f"hour={received:%H}")
    os.makedirs(folder, exist_ok=True)
    with open(os.path.join(folder, f"{site}.jsonl"), "a", encoding="utf-8") as f:
        f.write(json.dumps(row) + "\n")

client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id="lake-ingest-k4p9")
if os.getenv("MQTT_USERNAME"):
    client.username_pw_set(os.getenv("MQTT_USERNAME"), os.getenv("MQTT_PASSWORD"))
    client.tls_set()
client.on_connect = on_connect
client.on_message = on_message
client.connect(BROKER, PORT, keepalive=60)
client.loop_forever()

Then convert a day of landing files to Parquet with pandas (pip install pandas pyarrow):

import glob
import pandas as pd

files = glob.glob("./landing/uns/date=2026-10-16/hour=*/*.jsonl")
df = pd.concat((pd.read_json(f, lines=True) for f in files), ignore_index=True)
df.to_parquet("./curated/telemetry_2026-10-16.parquet", index=False)
print(df.groupby(["site", "line", "machine"])["temperature_c"].mean())

Point the landing directory at a folder that syncs to your object storage, or swap the file write for your cloud SDK's upload call. Test against the public MQTT test broker with simulated data first; it is readable by anyone, so real plant data belongs on a private broker with credentials.

Making Ingestion Reliable

A landing-zone subscriber is simple, which is good, but plan for these details:

1. Use a persistent session. With a fixed client ID, clean_session=False (MQTT 3.1.1) or a session expiry interval (MQTT 5), the broker queues QoS 1 messages during short ingestion outages.
2. Scale with shared subscriptions. On MQTT 5, several ingestion workers can share one subscription such as $share/lake/<prefix>/# so each message is processed once.
3. Record both timestamps. Keep the source timestamp from the payload and the receive time, so late or buffered data can be detected.
4. Do not rely on the broker for history. A broker's queue covers minutes of disconnection, not days. Long outages need buffering at the edge, such as store-and-forward.
5. Version the schema. Add a schema version field to payloads so the lake can handle changes.

On ConnectMQ, shared subscriptions are available on the paid Serverless plan and above, and the free plan supports QoS up to 1, which is what this ingestion pattern needs. ConnectMQ does not include built-in database or data lake bridges; the honest and flexible pattern is exactly this one, an external subscriber you control.

Which Should You Build First?

If you have neither, start with the namespace for one line, then add storage as the first consumer. Real-time visibility delivers value quickly and gives the lake clean, contextualised input later. If you already have a lake fed by batch exports from SCADA or MES, adding a UNS gives you fresher data and removes one-off connectors; point the lake's ingestion at the namespace and retire the exports one by one.

For the wider architecture, our industrial IoT MQTT guide shows where both fit alongside PLCs, gateways and SCADA.

Connect Your Unified Namespace to Storage With ConnectMQ

The namespace only feeds the lake reliably if the broker in the middle is dependable. ConnectMQ is a cloud MQTT broker with MQTT 3.1.1 and 5.0, TLS, retained messages and shared subscriptions on paid plans, plus ISA-95 topic detection in the portal. Create a broker on the free Serverless plan at app.connectmq.com, no credit card required, and run your ingestion subscriber against it today.

Frequently Asked Questions

Can a unified namespace replace a data lake?

No. A UNS holds current state and streams events; it is not designed for long-term storage or analytical queries. It works best as the source that feeds a lake.

Can a data lake replace a unified namespace?

Not for real-time use. Lakes are optimised for large historical queries, not for pushing live values to HMIs, alerting and other machines within seconds.

How do I get MQTT data into a data lake?

Run a subscriber, such as a Python script, Telegraf or Node-RED, that consumes namespace topics and writes files or rows to your storage. Partition by date and keep ISA-95 levels as columns.

Should the historian and the data lake both subscribe to the UNS?

Often yes. The historian serves operators with recent high-resolution trends, while the lake keeps long-term data for analytics and machine learning.