Index messages locally in an encrypted on-device store so encrypted rooms can be searched at all

[hayaksi1]

Content

Problem

Element X cannot search messages in encrypted rooms. The server-side /search endpoint only ever
sees ciphertext, so the one search path the app has is structurally unable to answer for E2EE
rooms — it returns nothing rather than reporting that it cannot help. The visible symptom is
#4149: the in-room search option is present in unencrypted rooms and
absent in encrypted ones. The workaround users describe ends in searching from another client
instead (#7072).

The Rust SDK now ships a local, encrypted, tantivy-backed message index — the matrix-sdk-search
crate tracked by matrix-org/matrix-rust-sdk#5350 — but nothing in this app has ever wired it
up. This adds the matrix-layer plumbing to do so.

Fix

• MatrixClient gains isMessageSearchAvailable and messageSearchService.
  RustMatrixClientFactory decides availability once, from
  featureFlagService.isFeatureEnabled(MessageSearch) && clientSecret != null, and calls
  withSearchIndexStore(path = <fileDir>/search-index, password = clientSecret) on the
  ClientBuilder. A session with no client secret gets no index — see Scope.
• A new libraries/matrix/api/.../search/ package keeps the SDK out of the UI, per the wrapping
  convention: MessageSearch is one stateful cursor exposing
  results: StateFlow<ImmutableList<MessageSearchResult>> and
  paginationState: StateFlow<MessageSearchPaginationState>, with setQuery and paginate
  returning Result. Its lifetime is the CoroutineScope that created it, so there is no
  close() to forget.
• RustMessageSearch funnels the SDK's non-suspend listener batches through a single
  Channel consumer so diffs stay ordered, and disposes its TaskHandles via
  scope.coroutineContext.job.invokeOnCompletion. MessageSearchResultsProcessor applies
  SearchServiceResultsUpdate diffs under a mutex against a list kept index-parallel with the
  SDK's, because the diffs are positional. Both it and MessageSearchResultMapper use an
  exhaustive when with no else, so an SDK bump fails compilation rather than silently
  dropping a variant.
• TantivyQueryEscaper escapes query syntax so what the user types is searched as text. Today
  a query containing :), a stray ", or the word AND is parsed as tantivy query syntax and
  can fail the whole search. Escaping runs in a single forward pass so added backslashes are not
  re-escaped, and the + Must prefix is applied after escaping — prefixing first would emit a
  literal \+ that parses cleanly and silently restores OR semantics.
• getCacheSize() now counts the index directory, and disabling the flag deletes it.

Everything is behind FeatureFlags.MessageSearch (feature.message_search,
defaultValue = { false }, isFinished = false). With the flag off — the default — no index
store is attached, no index directory is created, and no other code path changes.

Motivation and context

Relates to #3709 (search messages in room) — the canonical tracker,
and the thread that argues for this design.
Relates to #6557 (in-room keyword search with timestamps and
jump-to-message).
Relates to #4149 (closed as a duplicate of #3709; a defect report,
with screenshots).
Relates to #7072.

None of these is closed here — the change ships behind a disabled feature flag, so they are cited
as context rather than as fixes.

Screenshots / GIFs

There is no UI change in this pull request.

Tests

Covered by unit tests rather than by manual steps.

TantivyQueryEscaperTest — expectations checked against a real tantivy 0.26.1 parser, covering
each escaped character in leading and non-leading position, the bare-word operators
(AND/OR/NOT/IN/TO), the no-re-escaping property, and the deliberate case where a token
with no letter or digit is left optional because the default tokenizer indexes no terms for it.

MessageSearchResultsProcessorTest — every SearchServiceResultsUpdate variant:
Append, PushBack, PushFront, Set, Insert, Remove, Truncate, Reset, PopBack, PopFront and Clear,
asserting the local list stays index-parallel with the SDK's.

RustMessageSearchTest — lazy subscription on first setQuery; that setQuery resets
paginationState to Idle(endReached = false) before results arrive, since a fresh SDK cursor
reports endReached = true; ordered delivery of batched listener callbacks; and handle disposal
when the scope completes.

RustMatrixClientFactoryTest — the index store is attached when the flag is on and a client
secret exists, and not attached when either is missing.

New fakes for downstream modules: FakeMessageSearch, FakeMessageSearchService,
FakeFfiSearchService.

Scope

Deliberately not attempted here, so it is not mistaken for an oversight:

• Search is entirely local. The on-device index only; there is no server-side /search
  fallback. Anything the index never saw is unfindable.
• Every query word is mandatory. Each escaped token becomes a + Must clause, so one typo
  returns zero results where an OR query would have degraded to partial matches. The trade-off
  is deliberate — OR over a local index returned mostly noise — but it is a trade-off.
• Matching is whole-word and unstemmed. The * prefix operator is escaped, so partial words
  do not match, and photo does not match photos. Tokenisation is fixed when the index is
  created and is effectively English-first, so a script the default tokenizer does not segment is
  served poorly; both are worth their own follow-ups rather than a fix buried here.
• Deliberate query syntax no longer works — "phrase", -exclude, ranges, regex and field
  lookups are all escaped to literal text. None of it was advertised in the UI, and malformed
  use of it used to kill the whole search.
• What the SDK indexes is opaque from here. Indexing is a side effect of events passing
  through the SDK's event cache. Media and sticker indexing, edit semantics, and
  removal-on-redaction — currently broken upstream on Meta: Full text search support matrix-org/matrix-rust-sdk#5350 — are all
  SDK-side.
• Sessions with no client secret never get an index and search is silently unavailable to
  them: legacy pre-schema-v5 sessions, and sessions created by a release build between
  2024-01-18 and 2024-04-15.
• The flag is read at client-build time only, so toggling it mid-session does nothing until
  restart.
• Sender filtering and space scoping are out of scope; happy to file linked tracking issues if
  useful.

Notes for reviewers

First of four. The rest, each depending only on this one: the search screen and its top-bar
entry point; the background history backfill; and the event-cache coverage bootstrap. Since a
pull request here must be based on a branch in this repository, they follow serially rather than
as a stack.

This is ~1,430 insertions, above the 500 guideline. Roughly half is tests and fakes, and
splitting the tantivy escaping or the API definition away from the implementation they cover
would make review harder rather than easier — but if you would prefer TantivyQueryEscaper and
its test as a separate ~200-line change, say so and I will pull it out.

Tested devices

• Physical
• Emulator
• OS version(s): Android 16 (API 36), Pixel 10 Pro emulator (arm64-v8a)

Checklist

• I am aware of the etiquette.
• This PR was made with the help of AI:
  • Yes. In this case, please request a review by Copilot.
  • No.
• Changes have been tested on an Android device or Android emulator with API 24
• UI change has been tested on both light and dark themes
• Accessibility has been taken into account. See https://github.com/element-hq/element-x-android/blob/develop/CONTRIBUTING.md#accessibility
• Pull request is based on the develop branch
• Pull request title will be used in the release note, it clearly defines what will change for the user
• Pull request includes screenshots or videos if containing UI changes
• You've made a self review of your PR

[CLAassistant]

All committers have signed the CLA.

[github-actions]

Thank you for your contribution! Here are a few things to check in the PR to ensure it's reviewed as quickly as possible:

• If your pull request adds a feature or modifies the UI, this should have an equivalent pull request in the Element X iOS repo unless it only affects an Android-only behaviour or is behind a disabled feature flag, since we need parity in both clients to consider a feature done. It will also need to be approved by our product and design teams before being merged, so it's usually a good idea to discuss the changes in a Github issue first and then start working on them once the approach has been validated.
• Your branch should be based on origin/develop, at least when it was created.
• The title of the PR will be used for release notes, so it needs to describe the change visible to the user.
• The test pass locally running ./gradlew test.
• The code quality check suite pass locally running ./gradlew runQualityChecks.
• If you modified anything related to the UI, including previews, you'll have to run the Record screenshots GH action in your forked repo: that will generate compatible new screenshots. However, given Github Actions limitations, it will prevent the CI from running temporarily, until you upload a new commit after that one. To do so, just pull the latest changes and push an empty commit.

[sanyamseac]

@jmartinesp @ganfra Is there any plan to merge this pr anytime soon, or will it take time to review and test?

[jmartinesp]

@jmartinesp @ganfra Is there any plan to merge this pr anytime soon, or will it take time to review and test?

We're about to start working in this feature on Android, and while we appreciate the conrtibutions, the problem is some of the PRs clash with our strategy (like adding the logic on Rust instead of Android) or designs, so we'd rather build those ourselves. Also, merging huge AI-generated PRs has come back to bite us in the past, so we'd rather proceed with caution with this feature.

That said, using this PR as a base seems fine: it looks like it provides some nice scaffolding for the feature. Let's see if we can make it mergeable.

[jmartinesp]

Odd, the failing CI flows pass locally, and a rebase didn't seem to help. It also happened in a different PR.

[jmartinesp]

After a quick look, I have a few questions and possible improvements to the code.

[jmartinesp]

Thanks for the changes, I think the code looks good now! Don't mind the Konsist tests, I think I know where they're coming from and there's no way they could work for forks, so I'll just force merge this.

[jmartinesp]

Actually, the PR that should fix the issues with Konsist was just merged, so let's use this PR to test it works as expected for fork PRs.

[codecov]

Codecov Report

❌ Patch coverage is 72.33010% with 57 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.54%. Comparing base (af9db20) to head (aefc595).

Files with missing lines	Patch %	Lines
.../libraries/matrix/test/search/FakeMessageSearch.kt	23.68%	29 Missing ⚠️
...d/libraries/matrix/impl/RustMatrixClientFactory.kt	56.25%	5 Missing and 2 partials ⚠️
...ies/matrix/impl/search/RustMessageSearchService.kt	30.00%	7 Missing ⚠️
...atrix/impl/auth/RustMatrixAuthenticationService.kt	37.50%	5 Missing ⚠️
...ies/matrix/test/search/FakeMessageSearchService.kt	16.66%	5 Missing ⚠️
...ibraries/matrix/api/search/MessageSearchService.kt	0.00%	1 Missing ⚠️
.../android/libraries/matrix/impl/RustMatrixClient.kt	80.00%	1 Missing ⚠️
...atrix/impl/search/MessageSearchResultsProcessor.kt	97.05%	0 Missing and 1 partial ⚠️
.../libraries/matrix/impl/search/RustMessageSearch.kt	97.91%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #7249      +/-   ##
===========================================
- Coverage    80.56%   80.54%   -0.02%     
===========================================
  Files         2758     2768      +10     
  Lines        80310    80510     +200     
  Branches     10951    10975      +24     
===========================================
+ Hits         64698    64844     +146     
- Misses       11379    11428      +49     
- Partials      4233     4238       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.