Flaky test report: committed-code failures on 2026-06-15

[andrross]

Flaky test report: committed-code failures on 2026-06-15

This report covers gradle-check builds run against committed code (invoke_type=Timer on git_reference=main, or invoke_type=Post Merge Action with the canonical push trigger main PR title) in the 24 hours preceding 2026-06-15T10:00:05Z.

The metrics cluster (metrics.opensearch.org) recorded 2 distinct test failures matching this filter, in 2 separate builds. Both were Timer runs against main. No Post Merge Action failures were recorded in the window.

Summary table (sorted by historical unique builds affected)

Test	Build link	Local repro w/ seed	First seen	Total unique builds affected	Pattern
org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testSnapshotAndRestore	80439	Not attempted (env blocker — :test:fixtures:azure-fixture:composeBuild needs docker buildx ≥ 0.17, local has 0.12.1)	2024-03-28	142	Chronic, bursty, slowly improving overall
org.opensearch.client.ReindexIT.testReindexTask	80411	Did not reproduce (1 run, BUILD SUCCESSFUL)	2024-06-18	34	Chronic-low, recently worsening (Apr–Jun 2026 cluster)

Both failures are long-lived flakes, not new regressions. Neither appears tied to a recent code change in the test or its system-under-test.

1. AzureBlobStoreRepositoryTests.testSnapshotAndRestore

• Build that failed: gradle-check/80439 (Timer, main, 2026-06-15T07:10Z, build_result=UNSTABLE)
• Failure mode: AssertionError: CloseIndexResponse failed - not acked Expected: <true> but: was <false> at OpenSearchBlobStoreRepositoryIntegTestCase.testSnapshotAndRestore line 364 (assertAcked(client().admin().indices().prepareClose(...))).
• Seed: B32B088760D8EDA7
• Reproduce command (from the build's stdout):

  ./gradlew ':plugins:repository-azure:internalClusterTest' \
    --tests 'org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testSnapshotAndRestore' \
    -Dtests.seed=B32B088760D8EDA7 -Dtests.security.manager=true \
    -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" \
    -Dtests.locale=en-PG -Dtests.timezone=America/Kralendijk -Druntime.java=25
  

• Local reproduction: Not attempted. The internalClusterTest task depends on :test:fixtures:azure-fixture:composeBuild, which calls docker compose build and requires buildx >= 0.17.0. The local environment has buildx 0.12.1, so the fixture build fails before the test JVM ever starts. Log: /tmp/flaky-report-2026-06-15/repro-Azure.log. Per the assignment scope, no system tooling was upgraded.
• First seen: 2024-03-28T06:57Z.
• Total unique builds affected (all build types, lifetime): 142.
• Monthly unique-build counts (selected): 2024-08:14, 2024-09:19, 2024-12:7, 2025-09:7, 2025-12:8, 2026-03:8, 2026-06:2 (month-to-date).
• Pattern: Chronic with bursts. Peak activity was Aug–Sep 2024 (~14–19 affected builds/month). The flake has cooled considerably through 2025–2026 (most months in the 1–8 range), but it has never gone dormant for more than ~1 month at a time. It still fires against committed code occasionally (Timer/Post-Merge appearances in 2025-04, 2025-08, 2025-09, 2025-10, 2025-11, 2025-12, 2026-01, 2026-02, 2026-03, 2026-04, 2026-06). Direction: slowly improving overall but still active.

2. ReindexIT.testReindexTask

• Build that failed: gradle-check/80411 (Timer, main, 2026-06-15T03:10Z, build_result=UNSTABLE)
• Failure mode: bare AssertionError at OpenSearchRestHighLevelClientTestCase.checkTaskCompletionStatus line 365, called via OpenSearchTestCase.assertBusy from ReindexIT.testReindexTask line 154. Seed in stack: [B1BA371696D8F363:29103A19A04D19FF]. This is a polling timeout in assertBusy, i.e. the task didn't reach the expected state inside the poll budget.
• Seed: B1BA371696D8F363
• Reproduce command (from the build's stdout):

  ./gradlew ':client:rest-high-level:asyncIntegTest' \
    --tests 'org.opensearch.client.ReindexIT.testReindexTask' \
    -Dtests.seed=B1BA371696D8F363 -Dtests.security.manager=true \
    -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" \
    -Dtests.locale=mg-MG -Dtests.timezone=Asia/Kuala_Lumpur -Druntime.java=25
  

• Local reproduction: Did not reproduce. Single run with the original seed against the local repo (HEAD 90fdbf95a0f, which is one commit ahead of the failing build's main) completed with BUILD SUCCESSFUL in 33s. Log: /tmp/flaky-report-2026-06-15/repro-Reindex.log. As the steering notes call out, RandomizedRunner seeds the Random streams but does not control thread scheduling, network timing, or assertBusy polling timing, so assertBusy-based timeouts typically don't reproduce deterministically from the seed alone.
• First seen: 2024-06-18T22:50Z.
• Total unique builds affected (all build types, lifetime): 34.
• Monthly unique-build counts: 2024-06:4, 2024-07:1, 2024-08:1, 2024-09:1, 2024-10:1, 2024-11:2, 2025-01:1, 2025-03:1, 2025-06:1, 2025-07:1, 2025-10:2, 2025-12:2, 2026-02:1, 2026-03:1, 2026-04:3, 2026-05:8, 2026-06:3 (month-to-date).
• Pattern: Low-rate chronic flake (~1–4 builds/month) for most of its life, almost entirely on PR builds, but the failure rate has clearly stepped up since April 2026 (8 unique builds in May 2026 alone) and the test has now started firing on Timer/Post Merge Action runs (1 each in April, 1 in May, 1 in June). The shift from PR-only to also hitting committed-code lanes is consistent with the April 2026 CI runner change from m5.8xlarge to m7a.8xlarge — faster cores can amplify timing-sensitive assertBusy flakes either by changing relative scheduling or by making the test framework's "is this stable yet?" budget effectively tighter. Direction: worsening.

Notes on the data

• Field-mapping caveat: in this metrics index, test_class and test_name accept term queries directly (no .keyword subfield) but test_status only matches via the analyzed text field; both quirks differ slightly from the documented schema and were worked around in queries.
• Logs and raw query outputs are saved under /tmp/flaky-report-2026-06-15/ on the host where this run was performed.
• No code changes were made to the repository as part of this investigation.