Skip to content

build: migrate type checker from mypy to basedpyright#1170

Open
vringar wants to merge 3 commits into
masterfrom
typing/query-db-overloads
Open

build: migrate type checker from mypy to basedpyright#1170
vringar wants to merge 3 commits into
masterfrom
typing/query-db-overloads

Conversation

@vringar

@vringar vringar commented May 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Migrates OpenWPM's static type checker from mypy to basedpyright and resolves the genuine type errors that surfaced in the process. Organised as three commits.

1. feat(typing): overload query_db and get_javascript_entries

db_utils.query_db (and the wrapper get_javascript_entries) returns list[sqlite3.Row] by default and list[tuple[Any, ...]] only with as_tuple=True. The old union return type forced every default-mode caller to add assert isinstance(row, sqlite3.Row) or # type: ignore. @overload signatures now make the return type follow the as_tuple keyword:

call static return type
query_db(db, q) list[sqlite3.Row]
query_db(db, q, as_tuple=False) list[sqlite3.Row]
query_db(db, q, as_tuple=True) list[tuple[Any, ...]]
query_db(db, q, as_tuple=some_bool) list[sqlite3.Row] | list[tuple[Any, ...]]

as_tuple is now keyword-only — the overloads would otherwise overlap. Redundant isinstance assertions in the test suite are removed.

2. build: migrate type checker from mypy to basedpyright

  • mypy replaced with basedpyright in pyproject.toml, .pre-commit-config.yaml, and the conda environment files
  • typeCheckingMode = "recommended"
  • a multiprocess type stub added so the untyped library stops producing false positives

3. fix(typing): resolve type errors surfaced by basedpyright

Genuine bugs and type errors basedpyright caught:

  • a possibly-unbound browser_profile_path in BrowserManager
  • missing None-checks before open() / screenshot paths
  • an exception passed where traceback.format_exc() expected a frame limit
  • an incompatible store_blob override in the in-memory storage provider, realigned to the abstract signature
  • precise Queue payload types on the BrowserManager / StorageController IPC queues
  • the storage base classes marked explicitly ABC

Also deletes the unmaintained build_cookie_table utility and the cookie module it was the sole importer of — both depended on the long-uninstallable netlib package and were already un-importable.

The remaining recommended-mode findings (untyped third-party libraries, intentional bad-input tests, the manual_test.py debug script) are frozen in .basedpyright/baseline.json, so CI gates only on newly introduced issues.

Test plan

  • basedpyright — 0 errors, 0 warnings
  • pre-commit run --all-files passes
  • full pytest suite — 137 passed, 3 skipped. The lone failure, test_audio_fingerprinting, is the known pre-existing flake (@pytest.mark.skipif(CI), "Flaky on CI") and is unrelated to these changes.
  • CI

Copilot AI review requested due to automatic review settings May 9, 2026 21:54

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves static typing for openwpm.utilities.db_utils.query_db and its wrapper get_javascript_entries by using @overload so the return type is determined by the as_tuple argument (rows by default, tuples only when as_tuple=True), without changing runtime behavior.

Changes:

  • Add @overload signatures for query_db to precisely type as_tuple=False vs as_tuple=True.
  • Apply the same overload pattern to get_javascript_entries.
  • Update typing imports to support overload/Literal.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread openwpm/utilities/db_utils.py Outdated
Comment thread openwpm/utilities/db_utils.py Outdated
@codecov

codecov Bot commented May 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 68.00000% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.12%. Comparing base (06d755a) to head (78fdbe4).

Files with missing lines Patch % Lines
openwpm/commands/utils/XPathUtil.py 0.00% 8 Missing ⚠️
test/test_mp_logger.py 0.00% 2 Missing ⚠️
test/test_profile.py 0.00% 2 Missing ⚠️
openwpm/browser_manager.py 75.00% 1 Missing ⚠️
openwpm/config.py 83.33% 1 Missing ⚠️
openwpm/js_instrumentation.py 0.00% 1 Missing ⚠️
openwpm/socket_interface.py 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1170      +/-   ##
==========================================
+ Coverage   62.19%   68.12%   +5.92%     
==========================================
  Files          40       38       -2     
  Lines        3902     3576     -326     
==========================================
+ Hits         2427     2436       +9     
+ Misses       1475     1140     -335     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vringar vringar force-pushed the typing/query-db-overloads branch from ed4a23b to a7989aa Compare May 10, 2026 00:10
@vringar vringar requested a review from Copilot May 10, 2026 14:11

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

Comment thread openwpm/utilities/db_utils.py
Comment thread openwpm/utilities/db_utils.py
@vringar vringar force-pushed the typing/query-db-overloads branch from a7989aa to bdb4cbd Compare May 12, 2026 14:14
@vringar vringar requested a review from Copilot May 12, 2026 22:22

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

openwpm/utilities/db_utils.py:68

  • Same overload ambiguity as query_db: as_tuple: Literal[True] = ... makes the True overload appear callable without specifying as_tuple, overlapping with the default-False overload. Prefer making as_tuple=True required in this overload (no default) so inference can’t accidentally select the tuple-returning overload for default calls.
@overload
def get_javascript_entries(
    db: Path, all_columns: bool = ..., as_tuple: Literal[True] = ...
) -> List[Tuple[Any, ...]]: ...

Comment on lines +21 to +24
@overload
def query_db(
db: Path, query: str, params: Any = ..., as_tuple: Literal[True] = ...
) -> List[Tuple[Any, ...]]: ...
vringar added 3 commits May 17, 2026 12:14
Add @overload signatures so the return type (tuples vs sqlite Row) is inferred from the as_tuple keyword argument. Drop the now-redundant isinstance assertions in tests that the refined overloads make unnecessary.
Replace mypy with basedpyright in pyproject.toml, the pre-commit config, and the conda environment files. Add a multiprocess type stub so the untyped library stops producing false positives.
Fix genuine type errors found when switching to basedpyright: a possibly-unbound variable, missing None-checks, an exception passed where a traceback limit was expected, and an incompatible store_blob override in the in-memory provider. Annotate the BrowserManager and StorageController queues with precise payload types and mark the storage base classes as explicitly abstract.

Delete the unmaintained build_cookie_table utility, which imported the long-uninstallable netlib package and was therefore already un-importable, along with the now-orphaned cookie module it was the sole importer of.

Record the remaining recommended-mode findings in a basedpyright baseline so CI gates only on newly introduced issues.
@vringar vringar force-pushed the typing/query-db-overloads branch from bdb4cbd to 78fdbe4 Compare May 17, 2026 14:04
@vringar vringar changed the title feat(typing): @overload query_db and get_javascript_entries build: migrate type checker from mypy to basedpyright May 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants