feat: add GCP monitoring module and alert policies for DE-04-T2 - #307
Merged
anhvdq merged 12 commits intoSep 16, 2026
Conversation
IsurangiGuniyangodage
marked this pull request as ready for review
August 21, 2026 05:45
Contributor
Author
|
@callmesoffie1811 and @anhvdq Please have a look! |
IsurangiGuniyangodage
marked this pull request as draft
August 21, 2026 05:47
IsurangiGuniyangodage
marked this pull request as ready for review
August 30, 2026 22:39
IsurangiGuniyangodage
marked this pull request as draft
August 30, 2026 23:01
Contributor
Author
|
I updated the PR. Please have a look @callmesoffie1811 and @anhvdq |
IsurangiGuniyangodage
marked this pull request as ready for review
August 31, 2026 03:10
anhvdq
requested changes
Sep 6, 2026
anhvdq
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for your works on this PR
I put a comment on how we collect the stats of the jobs, it would be better to implement in the RunContext instead of current try-catch block
Besides, the terraform code seems good to go
…iables (Woolworths)
anhvdq
reviewed
Sep 13, 2026
anhvdq
left a comment
Collaborator
There was a problem hiding this comment.
The changes looks good on the IGA and WW scrapers but seems like Aldi and Coles were not updated yet
Could you take a look and fix them?
Then all good to go
Signed-off-by: Quoc Anh <41427319+anhvdq@users.noreply.github.com>
anhvdq
approved these changes
Sep 16, 2026
anhvdq
left a comment
Collaborator
There was a problem hiding this comment.
I have fixed some issues with the format and try-catch for better practices
All good now
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
DE-04-T2 requires monitoring and alerting for scraper/ETL pipeline failures, following the GCP-native approach recommended in DE-04-T1. Pipeline failures currently go unnoticed unless manually checked.
What changed
discount-mate-infra/modules/monitoring: four log-based metrics (scraper_rows, scraper_run_failed, scraper_http_errors, scraper_block) and three alert policies (job failed, zero/near-zero rows with missing-data-as-breach, HTTP error-rate spike)environments/prod(enabledmonitoring.googleapis.com, addedmonitoring_alert_emailvariable)terraform fmtandterraform validateboth passscrape_run_finishedsummary (rows scraped, duration, HTTP status breakdown, retries), ascrape_blockevent on detected soft-blocks/403s, and ascrape_run_failedevent on unhandled exceptions — the data the log-based metrics above depend onNotes
Infrastructure and scraper-side logging are both complete and internally consistent — the alert policies now reference log fields the scrapers actually emit.
Verified locally: set up a proper Python 3.12 environment for the pipeline using
uv sync(per itspyproject.toml/uv.lock), then confirmed all three logging functions (emit_scrape_summary,emit_scrape_block,emit_scrape_run_failed) produce valid JSON with the exact fields the Terraform log-based metrics expect (rows_scraped,event,scraper,http_403, etc.) for both success and failure cases.terraform applyagainst live infra and fault-injection testing (confirming each alert policy fires correctly end-to-end) will be handled by Ben Van. Marking as ready for review.Update: Refactored scraper run-stats tracking into a new
RunStatsmodel attached toRunContext, per Ben's review feedback — applied consistently across all four scrapers (IGA, Woolworths, ALDI, Coles), removing the extra try/except layers.