Add REES46 recommendation pipeline and purchase-probability model

[ethanfuyz]

Summary

This PR adds the ML/Recommendation_system/Recommendation-2026-t2/ sub-project, containing the REES46 data-processing, feature-engineering, model-training and Top-10 recommendation workflow.

It also updates .gitignore to exclude:

ML/Recommendation_system/Recommendation-2026-t2/data/

The data directory contains approximately 16 GB of raw and processed datasets and should not be committed to Git.

What was added

Time-based dataset preparation

• Processed the 42.4M-event REES46 October dataset in memory-efficient chunks.
• Split events chronologically into:
  • Train: before 22 October 2019
  • Validation: 22–26 October 2019
  • Test: from 27 October 2019
• Retained cold-start users and verified that there is no time overlap between splits.

Contributor: SERAY MIRNAK GULSEVEN

User-product interaction aggregation

• Added feature2_dataagg.ipynb.
• Aggregated events by user-product pair.
• Generated view, cart and purchase counts.
• Added user/product coverage and cold-start checks.

Contributor: ISHANI SACHIN BHONGALE

Initial feature extraction

• Added extract.ipynb.
• Implemented the initial recency, session, user-activity, product-popularity and price features.

Contributor: ROHIT SRINIVAS SHIBINENI

Additional features and model training

• Added DiscountMate_additional_features_model.ipynb.
• Created leakage-safe snapshots using:
  • 7-day feature history
  • 2-day recent-interaction candidate window
  • 3-day future purchase label
• Generated 55 model features covering:
  • User-product behaviour
  • User activity
  • Product, category and brand popularity
  • Recency
  • Price
  • Historical conversion rates
• Compared Logistic Regression with two Histogram Gradient Boosting configurations.
• Handled class imbalance using negative sampling and sample weighting.
• Selected the model using validation NDCG@10, followed by PR-AUC.
• Calibrated the selected model’s probabilities and implemented per-user Top-10 ranking.

Contributor: Team Leader Ethan Fu

Exported model

The selected calibrated Logistic Regression model and its metadata are included under:

• ML/Recommendation_system/Recommendation-2026-t2/models/DiscountMate_REES46_purchase_probability_model.joblib
• ML/Recommendation_system/Recommendation-2026-t2/models/DiscountMate_REES46_purchase_probability_model_metadata.json

The model outputs purchase_probability, which is used to rank candidate products for each user and select the Top 10.

Test results

Metric	Result
PR-AUC	0.1301
ROC-AUC	0.8977
Log Loss	0.01849
Brier Score	0.00349
Candidate-pair coverage	11.71%
Overall Recall@10	11.35%
Candidate-covered Recall@10	96.88%
Overall HitRate@10	13.11%
Overall NDCG@10	0.09997

The ranking model performs well for products already present in the recent-interaction candidate set. The main limitation is candidate coverage, which can be improved later using collaborative filtering, similar products and popularity-based candidate generation.

Data availability

The data/ folder is excluded through .gitignore because of its size.

To reproduce the processed datasets:

1. Download or restore the REES46 source and time-split Parquet files under data/.
2. Run the notebooks to regenerate the aggregated and additional-feature datasets.
3. Run DiscountMate_additional_features_model.ipynb to train, evaluate and export the model.

Verification

• The complete additional-feature notebook was executed successfully.
• Train, validation and test feature datasets were generated without duplicate snapshot keys, missing values, NaN values or infinite values.
• The exported model was reloaded successfully and produced valid purchase probabilities.
• Model selection used validation data only; the test set was reserved for final evaluation.

Documentation

Detailed methodology, feature definitions, model parameters, evaluation results and usage instructions are available in:

ML/Recommendation_system/Recommendation-2026-t2/README.md

[ethanfuyz]

ready to merge