Modern Data Engineering with SQL and Python

Model customers, orders and events with keys and cardinalities.

Evidence: A schema that prevents orphan orders and represents repeated events.

Query related tables while controlling duplicate row multiplication.

Evidence: Expected row counts and totals match fixture answers.

Calculate latest attempts and rolling operational measures.

Evidence: Queries handle ties and multiple attempts deterministically.

Read CSV, JSON and columnar files; validate expected fields.

Evidence: Invalid rows are quarantined with a specific reason.

Implement paginated incremental reads from a mock service.

Evidence: A second run does not duplicate records already ingested.

Compare bounded batches with late and out-of-order events.

Evidence: A design explains watermarks and a replay strategy.

Separate raw, cleaned and serving datasets with traceable changes.

Evidence: Each output row can be traced to its source batch.

Test uniqueness, completeness, ranges and relationships.

Evidence: All seeded defects are detected without rejecting valid fixtures.

Create dependent tasks with retry and timeout behaviour.

Evidence: A failed task retries safely and downstream work waits.

Reprocess an earlier interval without corrupting current data.

Evidence: Counts remain stable after a deliberate repeat backfill.

Inspect query plans and compare partition choices.

Evidence: A measured comparison records runtime and data scanned.

Write contracts, ownership and operational instructions.

Evidence: A peer can run and recover the pipeline using the README.