Skip to content

feat: add GCP monitoring module and alert policies for DE-04-T2 - #307

Merged
anhvdq merged 12 commits into
DataBytes-Organisation:mainfrom
IsurangiGuniyangodage:feature/de-04-t2-scraper-monitoring
Sep 16, 2026
Merged

anhvdq merged 12 commits into
DataBytes-Organisation:mainfrom
IsurangiGuniyangodage:feature/de-04-t2-scraper-monitoring

Conversation

@IsurangiGuniyangodage

@IsurangiGuniyangodage IsurangiGuniyangodage commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Why

DE-04-T2 requires monitoring and alerting for scraper/ETL pipeline failures, following the GCP-native approach recommended in DE-04-T1. Pipeline failures currently go unnoticed unless manually checked.

What changed

  • Added discount-mate-infra/modules/monitoring: four log-based metrics (scraper_rows, scraper_run_failed, scraper_http_errors, scraper_block) and three alert policies (job failed, zero/near-zero rows with missing-data-as-breach, HTTP error-rate spike)
  • Slack (#alerts-scrapers) and email notification channels, per the implementation plan's redundancy recommendation
  • Wired the module into environments/prod (enabled monitoring.googleapis.com, added monitoring_alert_email variable)
  • terraform fmt and terraform validate both pass
  • Instrumented all four scrapers (IGA, Woolworths, ALDI, Coles) with structured JSON logging: each now emits a scrape_run_finished summary (rows scraped, duration, HTTP status breakdown, retries), a scrape_block event on detected soft-blocks/403s, and a scrape_run_failed event on unhandled exceptions — the data the log-based metrics above depend on
  • All four scraper files verified to compile cleanly under Python 3.12 (the pipeline's required version)

Notes

Infrastructure and scraper-side logging are both complete and internally consistent — the alert policies now reference log fields the scrapers actually emit.

Verified locally: set up a proper Python 3.12 environment for the pipeline using uv sync (per its pyproject.toml/uv.lock), then confirmed all three logging functions (emit_scrape_summary, emit_scrape_block, emit_scrape_run_failed) produce valid JSON with the exact fields the Terraform log-based metrics expect (rows_scraped, event, scraper, http_403, etc.) for both success and failure cases.

terraform apply against live infra and fault-injection testing (confirming each alert policy fires correctly end-to-end) will be handled by Ben Van. Marking as ready for review.

Update: Refactored scraper run-stats tracking into a new RunStats model attached to RunContext, per Ben's review feedback — applied consistently across all four scrapers (IGA, Woolworths, ALDI, Coles), removing the extra try/except layers.

@IsurangiGuniyangodage
IsurangiGuniyangodage marked this pull request as ready for review August 21, 2026 05:45
@IsurangiGuniyangodage

Copy link
Copy Markdown
Contributor Author

@callmesoffie1811 and @anhvdq Please have a look!

@IsurangiGuniyangodage
IsurangiGuniyangodage marked this pull request as draft August 21, 2026 05:47
@IsurangiGuniyangodage
IsurangiGuniyangodage marked this pull request as ready for review August 30, 2026 22:39
@IsurangiGuniyangodage
IsurangiGuniyangodage marked this pull request as draft August 30, 2026 23:01
@IsurangiGuniyangodage

Copy link
Copy Markdown
Contributor Author

I updated the PR. Please have a look @callmesoffie1811 and @anhvdq

@IsurangiGuniyangodage
IsurangiGuniyangodage marked this pull request as ready for review August 31, 2026 03:10

@anhvdq anhvdq left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your works on this PR
I put a comment on how we collect the stats of the jobs, it would be better to implement in the RunContext instead of current try-catch block

Besides, the terraform code seems good to go

Comment thread DE/ingestion-pipeline/scraper/coles/coles_product_scraper.py Outdated

@anhvdq anhvdq left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes looks good on the IGA and WW scrapers but seems like Aldi and Coles were not updated yet
Could you take a look and fix them?
Then all good to go

@anhvdq anhvdq left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have fixed some issues with the format and try-catch for better practices
All good now

@anhvdq
anhvdq merged commit 7c03791 into DataBytes-Organisation:main Sep 16, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants