A non-UTF-8 .graphifyignore silently loses its non-ASCII rules: errors="ignore" turns an exclusion into a pattern that matches nothing

[abhay-codes07]

Summary

.gitignore, .graphifyignore and $GIT_DIR/info/exclude are read with errors="ignore", which turns a mis-encoded byte into no byte. An ignore file that is not valid UTF-8 therefore loses its non-ASCII rules silently — the pattern survives as a shorter string that matches nothing, the directory is scanned, and nothing anywhere says so.

graphify/detect.py (0.9.44), both read sites:

for raw in ignore_file.read_text(encoding="utf-8-sig", errors="ignore").splitlines():   # _load_dir_own_ignore
for raw in info_exclude.read_text(encoding="utf-8-sig", errors="ignore").splitlines():  # _load_graphifyignore

A rule reading Orçamento/ saved in the host ANSI codepage is Or\xe7amento/ on disk. \xe7 is not valid UTF-8, so it is dropped and the effective pattern becomes Oramento/.

Reproduce

from pathlib import Path
from graphify.detect import detect

base = Path(tempfile.mkdtemp())
(base / ".graphifyignore").write_bytes("Orçamento/\n".encode("cp1252"))
(base / "Orçamento").mkdir()
(base / "Orçamento" / "contrato.py").write_text("x = 1", encoding="utf-8")
(base / "main.py").write_text("print('hi')", encoding="utf-8")

print(sorted(Path(f).name for f in detect(base)["files"]["code"]))

--- .graphifyignore saved as utf-8 ---
  raw bytes            : b'Or\xc3\xa7amento/\n'
  excluded as intended : True
  files scanned        : ['main.py']

--- .graphifyignore saved as cp1252 ---
  raw bytes            : b'Or\xe7amento/\n'
  excluded as intended : False
  files scanned        : ['contrato.py', 'main.py']
  *** the ignore rule silently did nothing ***

Why it matters

An ignore rule failing open is not a cosmetic bug. test_graphifyignore_matches_nfd_path_with_nfc_pattern already states the stakes in its own docstring, for the normalisation version of this same failure:

Without normalization the two compare unequal and the rule silently does nothing — the files get scanned, and docs/PDFs are sent to an LLM despite an explicit exclusion.

That is exactly what happens here, reached by a different route. Someone who writes Orçamento/ is trying to keep that directory out of the corpus, and out of the semantic pass.

Non-UTF-8 ignore files are not exotic on Windows: ANSI was Notepad's default for years, and Set-Content still writes the ANSI codepage unless -Encoding is passed. Accented directory names are common in Portuguese, Spanish, French and German projects — the same population _nfc normalisation was added for.

Two existing tests could never have caught it

test_graphifyignore_matches_nfc_path_with_nfd_pattern and its mirror write the ignore file with write_text and no encoding=, so on Windows they emit the locale codepage. One raises UnicodeEncodeError outright (the NFD form's combining cedilla, U+0327, has no cp1252 representation); the other writes cp1252 bytes the reader then mangles. Both have been failing on Windows, and the test job runs ubuntu-latest only.

Suggested fix

Prefer UTF-8, but never silently drop a rule: decode UTF-8 (BOM-tolerant) first, and only on failure fall back to the host encoding and then latin-1, which cannot fail and maps every byte to a codepoint. A rule spelled in some third encoding still comes out wrong, but it comes out whole, and a one-time warning naming the file makes it fixable rather than invisible. Decoding must still never raise — a scan should not die on a stray byte in an ignore file.

I have this working with tests and will open a PR shortly.

Environment

graphify	v8 @ 4fca621 (0.9.44)
Python	3.12, Windows 11

[abhay-codes07]

Opened #2799 with the fix and regression coverage.

[safishamsi]

Fixed in v0.9.46 (on PyPI now) via #2799, thanks @abhay-codes07. Closing.