Skip to content

Decode text in the database encoding in SqliteDataReader - #39547

Merged
cincuranet merged 1 commit into
dotnet:mainfrom
officialasishkumar:fix/39546-sqlite-utf16-text-reader
Nov 6, 2026
Merged

cincuranet merged 1 commit into
dotnet:mainfrom
officialasishkumar:fix/39546-sqlite-utf16-text-reader

Conversation

@officialasishkumar

@officialasishkumar officialasishkumar commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #39546

GetTextReader, GetFieldValue<TextReader> and GetChars wrap GetStream in a StreamReader that decodes UTF-8. GetStream exposes text in the database encoding, because neither sqlite3_blob_read nor sqlite3_column_blob converts it. In a database created with PRAGMA encoding = 'UTF-16le' or 'UTF-16be', these methods therefore return text like t\0�\0s\0t\0 without an error, while GetString returns the right value.

Fix

SqliteDataRecord now creates its text reader in one place, used by GetTextReader and GetChars:

  • For a TEXT value in a database that isn't UTF-8, it reads the value with GetString, because sqlite3_column_text always converts to UTF-8. I chose this over a StreamReader with the database encoding because the bytes of a value that isn't read through a blob depend on earlier calls: once GetString has read a value, sqlite3_column_blob returns it as UTF-8, so the database encoding wouldn't reliably describe them.
  • In every other case nothing changes. UTF-8 databases keep streaming (GetTextReader_works and GetTextReader_works_streaming are unchanged), and BLOB values are still decoded as UTF-8.

The encoding comes from PRAGMA encoding, which runs at most once per result set and only when a TEXT value is read this way. All databases on a connection share one encoding, because SQLite rejects attaching a database with a different one.

GetStream and GetBytes still return the raw bytes.

Note that on UTF-16 databases, these methods now hold the whole value in memory rather than streaming it.

Tests (SqliteDataReaderTest)

  • GetTextReader_works_for_database_encoding and GetChars_works_for_database_encoding: UTF-16 LE and BE, with a rowid (blob path), without one, and from an expression, plus UTF-8 as a guard.
  • GetTextReader_works_for_parameter_when_UTF16_database and GetFieldValue_of_TextReader_works_when_UTF16_database.
  • GetTextReader_reads_blob_as_UTF8_when_UTF16_database: guards that BLOB values keep their behavior.

Without the source change, 12 of these test cases fail. The UTF-8 and BLOB guards pass either way. With it, the Microsoft.Data.Sqlite.Tests project passes on macOS arm64 (829 passed, 1 skipped). I only ran that test project locally, not the full repository suite.

  • I've read the guidelines for contributing and seen the walkthrough
  • I've posted a comment on an issue with a detailed description of how I am planning to contribute and got approval from a member of the team
  • The code builds and tests pass locally (also verified by our automated build checks)
  • Commit messages follow this format:
        Summary of the changes
        - Detail 1
        - Detail 2

        Fixes #bugnumber
  • Tests for the changes have been added (for bug fixes / features)
  • Code follows the same patterns and style as existing code in this repo

GetTextReader, GetFieldValue<TextReader> and GetChars wrapped GetStream in
a UTF-8 StreamReader. GetStream exposes text in the database encoding, so
in a UTF-16 database they silently returned garbage, while GetString
returned the right value.

- Read TEXT values through GetString, which SQLite always converts to
  UTF-8, when the database encoding isn't UTF-8
- Keep streaming for UTF-8 databases, and keep decoding BLOB values as
  UTF-8 in every database
- Query PRAGMA encoding once per result set, and only for TEXT values

Fixes dotnet#39546
@officialasishkumar
officialasishkumar requested a review from a team as a code owner November 6, 2026 04:48
Copilot AI balanced review requested due to automatic review settings November 6, 2026 04:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The UTF-16 text-decoding fix has no active review issues.

No open issues 🎉

What changed in this PR

Fixes #39546 by correcting text decoding in Microsoft.Data.Sqlite for UTF-16 databases across GetTextReader, GetFieldValue<TextReader>, and GetChars.

Changes:

  • Centralizes text-reader creation, using GetString for non-UTF-8 TEXT values and caching the database encoding per result set.
  • Preserves UTF-8 streaming and BLOB decoding; UTF-16 text is materialized in memory, while raw-byte APIs remain unchanged.
  • Adds regression tests for database encodings, query shapes, parameters, and BLOB behavior.
File Description
test/​Microsoft.Data.Sqlite.Tests/​SqliteDataReaderTest.cs Adds encoding regression tests and UTF-8/BLOB compatibility guards.
src/​Microsoft.Data.Sqlite.Core/​SqliteDataRecord.cs Shares encoding-aware text-reader creation between text and character accessors.

Review effort: Balanced

@cincuranet cincuranet self-assigned this Nov 6, 2026
@cincuranet
cincuranet merged commit b051f3d into dotnet:main Nov 6, 2026
21 checks passed
@dotnet-milestone-bot dotnet-milestone-bot Bot added this to the 12.0-preview1 milestone Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Microsoft.Data.Sqlite: GetTextReader and GetChars return garbage for TEXT values in a UTF-16 database

3 participants