Large Scale E Commerce Log Processing Pipeline with PySpark & Spark Architecture
Group: Capstone Project
|Product Category: Cloud & Data Engineering
|Sub Category: Apache Spark
About this Product
E-Commerce Log Processing Pipeline with PySpark is an advanced data engineering capstone project that builds a production-grade PySpark pipeline on a fictional e-commerce platform — ShopStream Analytics — processing 5GB of clickstream logs (scalable to 5TB) across 30M+ rows through a Bronze → Silver → Gold medallion architecture.
With this project, you'll build a pipeline that can:
- Ingest 30M+ clickstream events from Parquet files with predicate pushdown and column pruning
- Filter bot traffic and pipeline narrow transforms in a single Spark stage with zero disk writes
- Enrich logs via SortMerge joins (customers, orders) and a broadcast join (products — 2MB table)
- Compute 4 Gold analytics tables — category revenue, funnel analysis, customer engagement, and hourly traffic
- Verify Spark optimizations with explain(True) — pushdown filters, BroadcastHashJoin, whole-stage codegen
- Demonstrate fault tolerance by killing an executor mid-job and observing automatic task retry
This project teaches you:
- PySpark pipeline design — Bronze, Silver, and Gold medallion architecture
- Spark internals — DAG scheduling, stage boundaries, shuffle optimization, and Catalyst optimizer
- Join strategies — BroadcastHashJoin vs SortMergeJoin and why each is chosen
- Adaptive Query Execution (AQE), Tungsten code generation, and Kryo serialization
It uses Python, PySpark, Apache Spark, Parquet, Delta Lake, and YARN/Standalone cluster modes.
Why this project matters:
PySpark is a core skill in every data engineering role. This project teaches you to explain the Spark engine underneath the code — exactly what senior engineering interviews test for.
Resources
The complete dataset for the ShopStream Analytics pipeline project. Contains 4 Parquet files:
- Clickstream_logs — 600,000 user interaction events (PAGE_VIEW, SEARCH, ADD_TO_CART, PURCHASE) with device, browser, referrer, and bot flag columns. Primary fact table and core input to the Bronze ingestion stage.
- Orders — 40,000 purchase transactions with order amount, quantity, discount, payment method, and shipping details. SortMerge joined with clickstream logs in the Silver stage.
- Customers — 10,000 user profiles with demographics and customer segments (NEW, ACTIVE, LOYAL, VIP, CHURNED). SortMerge joined on user_id during Silver enrichment.
- Products — 200 product catalog entries with category, brand, price, and rating. The smallest table — purpose-built to demonstrate Spark's broadcast join optimization.
Project Mentors
Similar Products
Product Performance Dataset
Topics: SQL, PostgreSQL, Retail Performance
Basic Professional Data Analysis
Topics: SQL, PostgreSQL, Data Quality Analysis
Restaurant Performance & Menu Optimization
Topics: SQL, PostgreSQL, Data Analytics
Similar Services
Finding the best experts for you...
No Services Yet
Expert services for this product will appear here once available.
Top User Reviews
Loading reviews...
Be the first to review this product!
Please try refreshing the page.