E-commerce Data Pipeline

Context

A personal Data Engineering project using entirely synthetic e-commerce data, separate from the Spindox internship. The pipeline and dashboard are documented in a local repository.

Problem

Late orders, subsequent corrections and columns introduced in later runs can create duplicates or inconsistent business metrics unless ingestion and transformations share explicit rules.

Goal

Build repeatable loads, a dimensional model with an explicit grain and checks that make corrections and dashboard metrics verifiable.

My responsibilities

I built the synthetic generator, raw ingestion, dbt models, tests and Streamlit dashboard in the project repository. Business measures are calculated and checked in the marts, not recomputed in the interface.

Documented implementation

Python generates reproducible CSVs and loads them into DuckDB with a transactional manifest. dbt organizes staging, current-version selection and dimensional marts. An order-line fact connects to customer, product and date dimensions; Streamlit reads a snapshot in read-only mode.

Technical decisions and tradeoffs

• Deterministic upsert

  order_line_id is the stable key. The current version is selected by updated_at and, on a tie, a canonical payload comparison. This adds complexity but makes replay and arrival order irrelevant to the final state.

• Visible anomalies

  Economically invalid rows remain in the fact but do not contribute to revenue. Missing dimension keys point to technical Unknown members, retaining the fact without claiming the data is complete.

Checks and verification

Project tests document grain uniqueness, business rules, idempotent replay and parity between incremental and full-refresh results for the fact and dimensions.

Documented evidence and results

• Incremental load

  In the documented synthetic dataset, 10 new late rows and 5 corrections lead to 5,010 distinct order_line_id values, with no final duplicates.

• Checks

  Documented tests show idempotent replay and parity between incremental and full-refresh results for the fact and dimensions.

Limits and improvements

The workload is synthetic and local. It does not demonstrate behavior on real sources, production volumes or cloud platforms; the dashboard reads a snapshot and does not run ingestion on startup.

Technologies

• Python
• DuckDB
• dbt Core and dbt-duckdb
• SQL
• Streamlit