Data-Intensive Programming Course Exercises

This repository contains all exercise files from the COMP.CS.320 Data-Intensive Programming course. All exercises were implemented using Python and executed on Azure Databricks platform.

Course Overview

This course focused on processing large-scale data using distributed computing frameworks, primarily Apache Spark. The exercises covered fundamental concepts of data-intensive programming, including data manipulation, analysis, streaming, machine learning, and various data storage formats.

Technologies & Tools

• Platform: Azure Databricks
• Language: Python
• Framework: Apache Spark (PySpark)
• Storage: Azure Data Lake Storage Gen2 (ADLS Gen2)
• File Formats: CSV, Parquet, Delta Lake
• ML Library: Spark MLlib

What I Learned

The course covered data-intensive programming with Apache Spark on Databricks: DataFrame operations and aggregations, RDDs and text processing, time series data, Spark ML (linear regression, classification), structured streaming, and storage formats (CSV, Parquet, Delta Lake). The group assignment applied these in a larger project (sales and football analytics, optimization, ML, and image analysis).

Key Skills Acquired

1. Data Processing

• Efficient data loading from various sources (CSV, Parquet, Delta)
• Data transformation and cleaning
• Complex aggregations and window functions
• Handling missing values and NULL data

2. Distributed Computing

• Understanding Spark's distributed architecture
• Optimizing Spark operations (broadcasting, caching, partitioning)
• Avoiding unnecessary shuffles and expensive operations
• Performance tuning techniques

3. Data Storage

• Understanding different file formats and their use cases
• Schema management and evolution
• Delta Lake for ACID transactions on big data
• Efficient data storage strategies

4. Streaming Data

• Real-time data processing with Structured Streaming
• Handling streaming aggregations
• Managing streaming queries and output modes

5. Machine Learning

• Building ML pipelines with Spark ML
• Feature engineering and vector assembly
• Model training and evaluation
• Custom evaluation metrics

6. Advanced Analytics

• Time series analysis
• Text processing at scale
• Complex business logic implementation
• Multi-dataset joins and transformations

Optimization Techniques Learned

1. Schema Management: Using explicit schemas instead of inference for better performance
2. Column Pruning: Selecting only necessary columns early in the pipeline
3. Early Filtering: Filtering data as early as possible to reduce processing
4. Broadcasting: Using broadcast() for small datasets in joins
5. Avoiding Collect: Using first() instead of collect()[0] when only one value is needed
6. Efficient Aggregations: Combining multiple operations in single agg() calls
7. Window Functions: Using window functions for ranking and aggregations
8. Caching Strategy: Strategic use of caching for repeatedly used DataFrames

---

Course: COMP.CS.320 Data-Intensive Programming
Platform: Azure Databricks
Language: Python (PySpark)
Year: 2024-2025