Historians store time-series well but handle relational context (orders, batches, maintenance records) poorly; databases store context but not high-rate time-series. A data lake — a centralized repository holding raw and processed data in its native formats — is the modern answer to plant-wide analytics: one place where time-series, documents, images, and business data meet. But lakes built without architecture become data swamps. This article covers the patterns that keep an industrial lake useful.
What Belongs in an Industrial Data Lake
- Time-series process data — historian exports or direct OPC UA/MQTT streams: machine states, process values, energy, alarms.
- Context data — orders, batches, materials, quality results, maintenance records from MES, ERP, CMMS, and LIMS.
- Unstructured data — inspection images, maintenance photos, reports, manuals, training documents.
- Derived data — KPIs, models' features, forecasts, and analytics outputs.
The lake's promise is that any of these can be joined later for questions nobody planned for — which only works if the raw data lands unmodified with its original context.
Reference Architecture
A maintainable lake has three zones, not one bucket:
- Raw (bronze) zone — data exactly as received: full fidelity, immutable, timestamped with source and version. Nothing is ever edited here; bad data gets quarantined, never corrected.
- Curated (silver) zone — cleaned, aligned, and validated data: consistent units, corrected timestamps, joined assets. This is the zone analysts and engineers work from.
- Consumption (gold) zone — aggregates, KPI tables, feature stores, and report-ready datasets — the layer dashboards and ML consume directly.
The pipeline between zones is the real engineering: ingestion (from OPC UA, MQTT, historians, and system APIs), transformation (cleaning, alignment, unit conversion with explicit rules), and validation (schema checks, quality gates that stop or quarantine bad batches). Every transformation is versioned so a dashboard's numbers can be traced to a pipeline version.
Ingestion Patterns
- Historian export — periodic bulk export (e.g., nightly) of compressed historian data: simple, robust, but delays analytics by a day.
- Streaming ingest — MQTT/Sparkplug or OPC UA Pub/Sub into a message broker, then into the lake: near-real-time, more moving parts, needs buffering and replay design.
- API pull — REST/SQL pulls from MES, ERP, CMMS on schedule or on change: standard integration practice.
- File drop — CSV/parquet uploads from edge devices and laboratory systems: cheap, but must be schema-validated at the gate.
Most plants run a mix: streaming for critical machine data, bulk for historians, API for business systems, files for the long tail.
Time-Series Storage Choices
Time-series data in a lake is typically stored in columnar files (Parquet) partitioned by time and asset, served by an analytical engine (SQL-on-lake, or a dedicated time-series database for interactive querying). Practical guidance:
- Partition by date and asset group — queries on "last month, line 2" stay fast without exotic tuning.
- Store the original sampling rate for raw data (downsampled aggregates live in the gold zone).
- Keep the historian as the operational system of record; the lake is the analytical copy — a common, healthy split.
- Compression is significant for time-series (10–30× with columnar formats); budget storage accordingly.
Governance in the Lake
A lake without governance is a swamp within a year: duplicated tables with different names, pipelines nobody understands, and numbers that cannot be trusted. Minimum viable governance:
- A data catalog: every dataset with owner, source, refresh schedule, and quality status.
- Naming conventions and a schema registry with versioning — breaking schema changes must be visible to consumers.
- Access control per zone and per dataset (who may read raw, who may write gold).
- Retention rules matched to business and regulatory needs; industrial data is not kept forever by default.
- An annual review where datasets without owners or consumers are archived or deleted.
Analytics Consumption
The lake earns its cost when the gold zone feeds the consumers: KPI dashboards (replacing the spreadsheet-based reports), ad-hoc analysis by engineers (SQL over curated data), machine learning feature stores (versioned, reusable features — the ML pipeline's raw material), and regulatory reporting. Each consumer gets an interface contract; the lake absorbs the chaos of source systems behind that contract.
Summary
An industrial data lake is raw-plus-curated-plus-consumption zones with versioned pipelines between them, fed by streaming and bulk ingestion, governed by catalog, schema, access, and retention rules. It coexists with the historian (operational) rather than replacing it, and it pays off when the gold zone makes plant data available to dashboards, analysis, and ML without touching the source systems. Design the zones, validate at the gates, and govern from day one — the alternative is a swamp.