Skip to content

WIP: Add automatic content detection - #675

Draft
rubenwardy wants to merge 4 commits into
masterfrom
content-detection
Draft

WIP: Add automatic content detection#675
rubenwardy wants to merge 4 commits into
masterfrom
content-detection

Conversation

@rubenwardy

Copy link
Copy Markdown
Member

Fixes #483

Comment thread app/models/packages.py

content_path = db.Column(db.String(200), nullable=False)
match_path = db.Column(db.String(200), nullable=False)
confidence = db.Column(db.Float, nullable=False)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, I keep both phash and dhash seprate, it's good to have them both, right now the confidence is computed as (2*lower_of_two + higher_of_two)/3 - but we can tweak that later

Comment thread app/models/packages.py
conn.execute(stmt)


class PackageContentDetection(db.Model):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also here i don't see a field for dataset name, which below is called match_dataset.

Comment thread app/models/packages.py
created_at = db.Column(db.DateTime, nullable=False, default=datetime.datetime.utcnow)

hashes = db.relationship("ContentDetectionDatasetEntryHash", back_populates="dataset_entry",
lazy="dynamic", cascade="all, delete, delete-orphan")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe somewhat important, but reference hashes store orientation/flip too, e.g. [rot180_flip] - useful to know perhaps?

@ZenonSeth

Copy link
Copy Markdown
Contributor

General thought:

I was thinking about the usability on the website btw, for new releases.
Maybe what there should be is like a 'ok-list' for each package, which would be a list of hashes.

My thinking of the flow is like this:

  • New package gets submitted (or first scanned), everything below certain confidence threshold (since 0 is perfect match) gets flagged
  • A reviewer can go through and mark each as either "not applicable" - adding it to the 'ok-list' - OR - marking it as 'needs changes'
  • Then when a new version of the same mod gets submitted, things with hashes on the 'ok-list' get skipped, and everything else gets rechecked and re-added if necessary to reviewer list

@rubenwardy

rubenwardy commented Sep 2, 2026

Copy link
Copy Markdown
Member Author

Makes sense. That's the purpose of PackageContentDetection, there will be a state which could be something like new, ignored, or accepted

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Automatic copyright infringement detection (like ContentID)

2 participants