DE-06: Implement Coles product matching and deduplication - #296
Conversation
vidhipatel32
changed the title
anhvdq
left a comment
anhvdq
left a comment
There was a problem hiding this comment.
Thanks for your works on this PR.
Please help check and address all issues from my comments
There was a problem hiding this comment.
What is the purpose of this file? Is this just for documentation or it is used anywhere?
If it's not run in the code and only use for documentation, move it to docs folder and make it MD file instead of sql
Write it as a documentation file and you can mention the code in MD file using three backtick syntax
For example:
SELECT
COUNT(*) AS total_products,
COUNT(gtin) AS products_with_gtin,
COUNT(*) - COUNT(gtin) AS products_without_gtin,
ROUND(
100.0 * COUNT(gtin) / NULLIF(COUNT(*), 0),
2
) AS gtin_coverage_percent
FROM silver.dim_products;|
|
||
| CASE lower( | ||
| NULLIF( | ||
| regexp_extract( | ||
| lower(coalesce(raw_size, '')), | ||
| '^[0-9]+(?:\.[0-9]+)?\s*([a-z]+)', | ||
| 1 | ||
| ), | ||
| '' | ||
| ) | ||
| ) |
There was a problem hiding this comment.
Please check the similar changes on this file
The newline changes are redundant and create a lot of distraction / unnecessary diff view
If we don't have any actual change in the code, leave it as original and dont reformat the file
|
|
||
| WHEN 'kg' THEN 'kg' | ||
| WHEN 'kilo' THEN 'kg' | ||
| WHEN 'kilos' THEN 'kg' | ||
|
|
||
| WHEN 'ml' THEN 'ml' | ||
|
|
||
| WHEN 'l' THEN 'l' | ||
| WHEN 'lt' THEN 'l' | ||
| WHEN 'ltr' THEN 'l' | ||
| WHEN 'litre' THEN 'l' | ||
| WHEN 'litres' THEN 'l' | ||
| WHEN 'liter' THEN 'l' | ||
| WHEN 'liters' THEN 'l' | ||
|
|
||
| WHEN 'ea' THEN 'ea' | ||
| WHEN 'each' THEN 'ea' | ||
|
|
||
| WHEN 'pack' THEN 'pack' | ||
| WHEN 'pk' THEN 'pack' | ||
|
|
||
| WHEN 'm' THEN 'm' | ||
| WHEN 'metre' THEN 'm' | ||
| WHEN 'meter' THEN 'm' | ||
|
|
||
| ELSE NULL | ||
| END AS pack_uom, | ||
|
|
||
| COALESCE( | ||
| save_statement, | ||
| offer_description, | ||
|
|
||
| CASE | ||
| WHEN lower(coalesce(promotion_type, '')) NOT IN ('', 'everyday') THEN promotion_type | ||
| WHEN lower(coalesce(promotion_type, '')) | ||
| NOT IN ('', 'everyday') | ||
| THEN promotion_type | ||
| ELSE NULL | ||
| END, | ||
|
|
||
| CASE | ||
| WHEN lower(coalesce(special_type, '')) NOT IN ('', 'everyday') THEN special_type | ||
| WHEN lower(coalesce(special_type, '')) | ||
| NOT IN ('', 'everyday') | ||
| THEN special_type | ||
| ELSE NULL | ||
| END, | ||
|
|
||
| CASE | ||
| WHEN upper(coalesce(comparable, '')) = 'SPECIAL' THEN comparable | ||
| WHEN upper(coalesce(comparable, '')) = 'SPECIAL' | ||
| THEN comparable |
There was a problem hiding this comment.
Similar to previous comments, why do we add newlines?
Dont reformat the file
8127964 to
a8d93b6
Compare
anhvdq
left a comment
anhvdq
left a comment
There was a problem hiding this comment.
I made the changes for the issues on this ticket
Good to go now
2 participants
Summary
Implements DE-06 product matching and deduplication for the Coles ETL pipeline.
Changes
with price as the tie-breaker.
deduplication behaviour, limitations and validation evidence.
Validation
Coles pipeline executed successfully for:
2026-01-01to2026-05-04Results:
product identity combination.
Matching Strategy
The implementation uses deterministic matching based on normalised:
brand + product name + pack quantity + pack unitGTIN is documented as the preferred identifier when available, but GTIN
values are not populated in the current Coles Silver dataset, so GTIN-first
matching could not be directly validated.
Near-duplicate candidates are flagged for review rather than automatically
fuzzy-merged to avoid incorrect product identity merges.
Scope
This PR is limited to the Coles implementation and its validation/documentation.