Repository navigation
Decode text in the database encoding in SqliteDataReader - #39547
Merged
cincuranet merged 1 commit intoNov 6, 2026
Merged
cincuranet merged 1 commit into
cincuranet merged 1 commit into
Conversation
GetTextReader, GetFieldValue<TextReader> and GetChars wrapped GetStream in a UTF-8 StreamReader. GetStream exposes text in the database encoding, so in a UTF-16 database they silently returned garbage, while GetString returned the right value. - Read TEXT values through GetString, which SQLite always converts to UTF-8, when the database encoding isn't UTF-8 - Keep streaming for UTF-8 databases, and keep decoding BLOB values as UTF-8 in every database - Query PRAGMA encoding once per result set, and only for TEXT values Fixes dotnet#39546
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The UTF-16 text-decoding fix has no active review issues.
No open issues 🎉
What changed in this PR
Fixes #39546 by correcting text decoding in Microsoft.Data.Sqlite for UTF-16 databases across GetTextReader, GetFieldValue<TextReader>, and GetChars.
Changes:
- Centralizes text-reader creation, using
GetStringfor non-UTF-8TEXTvalues and caching the database encoding per result set. - Preserves UTF-8 streaming and BLOB decoding; UTF-16 text is materialized in memory, while raw-byte APIs remain unchanged.
- Adds regression tests for database encodings, query shapes, parameters, and BLOB behavior.
| File | Description |
|---|---|
test/Microsoft.Data.Sqlite.Tests/SqliteDataReaderTest.cs |
Adds encoding regression tests and UTF-8/BLOB compatibility guards. |
src/Microsoft.Data.Sqlite.Core/SqliteDataRecord.cs |
Shares encoding-aware text-reader creation between text and character accessors. |
Review effort: Balanced
cincuranet
approved these changes
Nov 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #39546
GetTextReader,GetFieldValue<TextReader>andGetCharswrapGetStreamin aStreamReaderthat decodes UTF-8.GetStreamexposes text in the database encoding, because neithersqlite3_blob_readnorsqlite3_column_blobconverts it. In a database created withPRAGMA encoding = 'UTF-16le'or'UTF-16be', these methods therefore return text liket\0�\0s\0t\0without an error, whileGetStringreturns the right value.Fix
SqliteDataRecordnow creates its text reader in one place, used byGetTextReaderandGetChars:TEXTvalue in a database that isn't UTF-8, it reads the value withGetString, becausesqlite3_column_textalways converts to UTF-8. I chose this over aStreamReaderwith the database encoding because the bytes of a value that isn't read through a blob depend on earlier calls: onceGetStringhas read a value,sqlite3_column_blobreturns it as UTF-8, so the database encoding wouldn't reliably describe them.GetTextReader_worksandGetTextReader_works_streamingare unchanged), andBLOBvalues are still decoded as UTF-8.The encoding comes from
PRAGMA encoding, which runs at most once per result set and only when aTEXTvalue is read this way. All databases on a connection share one encoding, because SQLite rejects attaching a database with a different one.GetStreamandGetBytesstill return the raw bytes.Note that on UTF-16 databases, these methods now hold the whole value in memory rather than streaming it.
Tests (
SqliteDataReaderTest)GetTextReader_works_for_database_encodingandGetChars_works_for_database_encoding: UTF-16 LE and BE, with a rowid (blob path), without one, and from an expression, plus UTF-8 as a guard.GetTextReader_works_for_parameter_when_UTF16_databaseandGetFieldValue_of_TextReader_works_when_UTF16_database.GetTextReader_reads_blob_as_UTF8_when_UTF16_database: guards thatBLOBvalues keep their behavior.Without the source change, 12 of these test cases fail. The UTF-8 and
BLOBguards pass either way. With it, theMicrosoft.Data.Sqlite.Testsproject passes on macOS arm64 (829 passed, 1 skipped). I only ran that test project locally, not the full repository suite.