The data engineer
in an agentic build

who is at the end of the pipe · the human who used to catch the error is gone

Ask the simplest question a room can ask about an agentic build, "what does the agent actually know?", and you are already asking a data-engineering question. That is the shift.

A data engineer's instruments assume a human at the end of the pipe. The consumer is an analyst, a dashboard, a report. Success is a pipeline that is fresh, complete, and correct: row counts match, the schema is valid, the job went green. The failure mode of bad data is a wrong number in a report, a sigh, and a manual patch. Nothing explodes, because a person is reading the output and can catch it.

Agentic work changes the consumer. It is now a model retrieving context at inference time and an agent taking actions on what it finds. Two things follow. First, the output the data engineer produces is no longer a table for a human to read; it is context for a machine to reason over, which means unstructured knowledge (documents, tickets, transcripts, wikis, code) becomes first-class alongside the warehouse. Second, the human shock-absorber is mostly gone: outside the checkpoints a build designs in, an agent acts on flawed or sensitive data at machine speed, before anyone can intervene. Poor data now produces a poorly performing agent.

That is why the centre of gravity moves from "is the pipeline fresh and correct?" to "can an agent find the right, authorised, trustworthy context at the moment it needs it, and can we prove it?" It is a more judgement-heavy job, not a lighter one.

the signature technical work · how the data is prepared is one of the largest levers on the answer

The new work has a shape: ingest, structure, index, serve, keep fresh. None of it is exotic. All of it is data engineering.

• Ingest. Turn documents into model-readable text. Naive text dumps lose tables and layout, and those errors cascade all the way to the answer, so layout-aware parsing is load-bearing. It is the closest new skill to existing ETL.
• Structure. Chunk the parsed text into retrievable units. Chunking is not a solved default: strategies trade recall against precision, and chunking inconsistently across a corpus quietly costs retrieval quality. Consistency here is a data-engineering discipline, the same instinct as a stable schema.
• Index, serve, keep fresh. Embed and store so chunks can be searched by meaning; expose retrieval through a governed interface with access enforced before the search runs; and re-index incrementally so a changed document becomes retrievable inside a defined window, and a deleted or reclassified one leaves the index just as fast. Orphaned chunks are both a quality bug and a leak. A stale index degrades answers silently, with no failed job to warn you.

the anti-hallucination move for structured data · the agent selects a definition; it can no longer invent one

The warehouse does not go away, and structured data has its own failure mode with agents. Let an agent write free-form SQL against raw tables and it can hallucinate joins, silently redefine a metric, and return different numbers for the same question asked different ways. The fix is not a better prompt. It is to constrain the agent to a governed semantic layer: metrics and dimensions defined once, in version-controlled form, so the agent selects from approved definitions and the layer generates the query.

controls at the data layer are the controls · a control you ask the model to honour is not a control

With an agent, three things follow, and each is a data-engineering responsibility: access travels with the chunk, the ingestion pipeline is a security boundary, and classification becomes the runtime gate on what an agent may see.

The through-line: end-to-end lineage from an agent's output back to the source data is what makes an agentic answer auditable and defensible to a regulator. That is the same lineage discipline the data engineer already practises, now doing heavier work. Where an organisation already maps its capabilities against data-classification levels, that mapping stops being a label on a shelf and becomes the runtime gate on what an agent can retrieve.

the fundamentals are not legacy · an agent amplifies whatever is underneath it

• Data quality used to fail into a bad report. Now it fails into an agent acting wrongly at speed. Higher stakes, same discipline.
• Data contracts enforced at source shift governance from auditing bad data after the fact to stopping it entering, the only speed that keeps up with an autonomous system.
• Modelling is what makes the semantic layer trustworthy. Good modelling is now the thing that stops an agent inventing numbers.
• Lineage and provenance are what make agentic answers auditable, and provenance is now a retrieval-time signal as well as an audit trail.
• The core craft (SQL, Python, orchestration, distributed processing, warehouse engineering) is still the daily toolkit, now also used to build parsing, chunking, embedding, and eval pipelines.

In summary: this is an upgrade of the craft. The old rules of data management are not being retired. They are being asked to hold up an autonomous system, which is a heavier load than a dashboard ever put on them.

a sequence you can walk · a strong data engineer is most of the way there already

The new skills build on what a data engineer already has rather than replacing it. Metadata catalogs become the foundation for context graphs, pipeline orchestration becomes agent-task design, data-quality monitoring becomes context-quality monitoring, and access control becomes governed agentic access. The order below is ours, and directional.

1. Retrieval literacyHow a model uses a context window, what retrieval is and why it works, embeddings and vector search as concepts. Enough to reason about the stack.
2. Unstructured-data engineeringDocument parsing, layout and table extraction, chunking and why consistency matters. Closest to existing ETL, so start here.
3. A retrieval pipeline end to endChunk, embed, store, retrieve. Build one from scratch; it is the single most concrete skill.
4. Retrieval quality engineeringHybrid search, rerankers, contextual retrieval, metadata filtering, and choosing the embedding model by testing it on your own corpus, not a leaderboard. Build a small scratch golden set the day the pipeline first runs: you cannot tune what you cannot score. Move from "it retrieves" to "it retrieves well."
5. The semantic layerGoverned metrics so agents query numbers instead of inventing them. Leans directly on existing modelling skill.
6. Evals and golden datasetsEngineer the golden set as a product: labelling workflow, versioning, contamination control, an eval gate wired into CI.
7. Secure and govern retrievalChunk-level access, isolation, sensitive-data handling, poisoning and injection defence, classification-aware retrieval.
8. AdvancedKnowledge graphs for multi-hop, agent memory, and the standard tool and context protocols that plug the retrieval layer into agents.

One gap is worth naming, because it is also an opportunity. As of mid-2026 there is no widely adopted, vendor-neutral competency framework for this role; the published learning paths are largely vendor tracks. Defining the standard by which an organisation grows its own agent-ready data engineers is exactly the kind of thing a Centre for Enablement exists to do.