Make the path from raw data to useful data visible.

Ingestion and modeling are connected jobs. Capture source data with enough context, then shape it into stable definitions people can use.

A simple data path

Keep the handoffs easy to explain.

Each stage should have a clear purpose, owner, and observable result. That makes failures easier to find and changes safer to review.

CaptureSources & events

LandRaw records

ShapeTyped, conformed data

ModelReusable entities

ServeMetrics & products

Ingestion choices

Choose a pattern that fits how data changes.

Batch ingestion is a good fit when updates arrive on a predictable schedule or near-real-time delivery is unnecessary. Change data capture can copy database changes as they occur. Event streaming helps systems react to a continuous sequence of events.

Those patterns have different requirements for ordering, replay, schema changes, latency, and operating complexity. Choose against a clear data freshness need rather than a tool trend.

Design promptHow fresh does the data need to be, and what should downstream users see if an upstream source is delayed?

# A useful contract describes the handoff
source: customer_events
owner: data_platform
format: jsonl
arrival: event_stream
key: event_id
expectations:
schema: versioned
late_data: retained
sensitive_fields: classified

Modeling principles

Keep each transformation layer purposeful.

Stage 1 · Source aligned

Staging

Rename, cast, and lightly clean fields so source details are consistent and easy to inspect.

Stage 2 · Reusable logic

Intermediate

Join and prepare shared business logic without hiding important assumptions.

Stage 3 · User facing

Marts

Present entities and measures in a shape that analysts and applications can use.

Add quality checks and traceability →