Flaky test report: committed-code failures on 2026-06-13

[andrross]

Flaky test report: committed-code failures on 2026-06-13

This report covers all distinct test failures seen in the OpenSearch metrics cluster
over the past 24 hours where the build was triggered against committed code
(invoke_type=Timer with git_reference=main, or invoke_type=Post Merge Action with
pull_request_title starting with "push trigger main"). Failures from PR builds are excluded
here as the source of the failure (because they may be PR-specific), but historical flake
rates do include all build types so we can see the underlying signal across time.

Methodology

• Detection window: 2026-06-12T10:00:00Z → 2026-06-13T10:00:06Z.
• Historical window: trailing 12 months, all invoke_type values (PR + Timer + Post Merge Action), aggregated by month with cardinality(build_number).
• Reproduction: each test was run locally one at a time against main at commit 90fdbf95a0f using the seed extracted from the failing build's stack trace (SeedInfo.seed([…])). For two suite-timeout failures whose listed test method is just the abandoned victim of a class-level timeout, the full test class was run.
• Local environment: dev desktop, 16 cores, JDK 25 runtime. Note that this is faster than CI runners; tests sensitive to more concurrency may reproduce more easily here, while tests sensitive to slower CI may reproduce less easily.
• Logs for each reproduction are saved under /tmp/flaky-report/logs/.

Summary table (sorted by historical unique builds affected)

#	Test	Builds (12M)	Committed-only builds	Reproduced locally?	Pattern
1	org.opensearch.versioning.ConcurrentSeqNoVersioningIT.testSeqNoCASLinearizability	113	31	no	worsening
2	org.opensearch.remotestore.RestoreShallowSnapshotV2IT.classMethod	105	29	yes (deterministic)	worsening
3	org.opensearch.remotestore.RestoreShallowSnapshotV2IT.testContinuousIndexing {p0=…}	61	24	no	stable
4	org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testShardPathDeletionWhenWarmIndexRelocate	23	12	no	worsening
5	org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=…}	12	2	no	stable
6	org.opensearch.repositories.s3.S3BlobContainerRetriesTests.classMethod	8	1	no	stable
7	org.opensearch.arrow.flight.bootstrap.tls.DefaultSslContextProviderFlightIT.testHotReloadWithNewInstance	7	3	no	newly observed
8	org.opensearch.repositories.s3.S3BlobContainerRetriesTests.testWriteLargeBlob	2	1	no	stable
9	org.opensearch.indices.replication.RemoteStoreReplicationSourceTests.testGetSegmentFiles	2	2	no	newly observed

Per-test detail

1. org.opensearch.versioning.ConcurrentSeqNoVersioningIT.testSeqNoCASLinearizability

• Recent failing build: gradle-check #80267 (D0E1AC8370763515:B2F82E96A6417A1)

• Original error: java.lang.AssertionError: ClusterHealthResponse has timed out - returned: [{"cluster_name":"TEST-TEST_WORKER_VM=[851]-CLUSTER_SEED=[2681696003229685198]-HASH=[4F81B92FF56]-cluster","status":"green","t

• First seen failing (12M window): 2025-06-28

• Most recent failure: 2026-06-12

• Total unique builds affected (12M): 113

• Committed-code-only unique builds (12M): 31

• By invoke type: Pull Request=77, Timer=29, Post Merge Action=7

• Pattern: worsening

• Monthly unique build counts:

  Month	Unique builds
  2025-06	1
  2025-07	4
  2025-08	4
  2025-09	2
  2025-10	6
  2025-11	5
  2025-12	4
  2026-01	3
  2026-02	10
  2026-03	3
  2026-04	35
  2026-05	25
  2026-06	11

• Local reproduction: Did not reproduce

  • Command: :server:internalClusterTest --tests <method>
  • Seed: D0E1AC8370763515:B2F82E96A6417A1
  • Duration: 18s, gradle exit code 0
  • Notes: Single method passed in 18s. Per repo notes, seed is not a reliable reproducer for this concurrent-disruption test.

2. org.opensearch.remotestore.RestoreShallowSnapshotV2IT.classMethod

• Recent failing build: gradle-check #80309 (D24BC247F931822D)

• Original error: java.lang.Exception: Suite timeout exceeded (>= 1200000 msec).

• First seen failing (12M window): 2025-06-20

• Most recent failure: 2026-06-13

• Total unique builds affected (12M): 105

• Committed-code-only unique builds (12M): 29

• By invoke type: Pull Request=75, Timer=22, Post Merge Action=8

• Pattern: worsening

• Monthly unique build counts:

  Month	Unique builds
  2025-06	4
  2025-07	8
  2025-08	4
  2025-09	4
  2025-10	10
  2025-11	8
  2025-12	10
  2026-01	9
  2026-02	7
  2026-03	6
  2026-04	13
  2026-05	14
  2026-06	8

• Local reproduction: Reproduced deterministically

  • Command: full class :server:internalClusterTest
  • Seed: D24BC247F931822D
  • Duration: 1213s, gradle exit code 1
  • Notes: Reproduced deterministically: same Suite timeout exceeded (>= 1200000 msec) at 20m12s, with same testContinuousIndexing as the abandoned test.

3. org.opensearch.remotestore.RestoreShallowSnapshotV2IT.testContinuousIndexing {p0={"opensearch.experimental.feature.writable_warm_index.enabled":"true"}}

• Recent failing build: gradle-check #80309 (D24BC247F931822D)

• Original error: java.lang.Exception: Test abandoned because suite timeout was reached.

• First seen failing (12M window): 2025-06-13

• Most recent failure: 2026-06-13

• Total unique builds affected (12M): 61

• Committed-code-only unique builds (12M): 24

• By invoke type: Pull Request=36, Timer=20, Post Merge Action=5

• Pattern: stable

• Monthly unique build counts:

  Month	Unique builds
  2025-06	2
  2025-07	11
  2025-08	8
  2025-09	2
  2025-10	5
  2025-11	5
  2025-12	3
  2026-01	3
  2026-02	5
  2026-03	4
  2026-04	5
  2026-05	6
  2026-06	2

• Local reproduction: Did not reproduce

  • Command: :server:internalClusterTest --tests <method>
  • Seed: D24BC247F931822D
  • Duration: 311s, gradle exit code 0
  • Notes: Single method passed in 5m11s. Original failure was "Test abandoned because suite timeout was reached" caused by cumulative class runtime; reproduced when running full class (see classMethod entry).

4. org.opensearch.indices.replication.WarmIndexSegmentReplicationIT.testShardPathDeletionWhenWarmIndexRelocate

• Recent failing build: gradle-check #80289 (341AAD0AE2E443C2:9A55C269057132C5)

• Original error: java.lang.AssertionError

• First seen failing (12M window): 2025-06-23

• Most recent failure: 2026-06-12

• Total unique builds affected (12M): 23

• Committed-code-only unique builds (12M): 12

• By invoke type: Pull Request=11, Timer=9, Post Merge Action=3

• Pattern: worsening

• Monthly unique build counts:

  Month	Unique builds
  2025-06	1
  2025-07	1
  2025-08	2
  2025-09	2
  2025-10	0
  2025-11	1
  2025-12	1
  2026-01	1
  2026-02	0
  2026-03	0
  2026-04	7
  2026-05	2
  2026-06	5

• Local reproduction: Did not reproduce

  • Command: :server:internalClusterTest --tests <method>
  • Seed: 341AAD0AE2E443C2:9A55C269057132C5
  • Duration: 15s, gradle exit code 0
  • Notes: Single method passed in 14s.

5. org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search.aggregation/90_sig_text_field_match_only_text/Dedup noise}

• Recent failing build: gradle-check #80292 (9CA5C4992680F376)

• Original error: java.lang.AssertionError: Failure at [search.aggregation/90_sig_text_field_match_only_text:152]: aggregations.class.buckets.0.sig_text.buckets.0.key didn't match expected value: aggregations.class.buc

• First seen failing (12M window): 2025-07-11

• Most recent failure: 2026-06-12

• Total unique builds affected (12M): 12

• Committed-code-only unique builds (12M): 2

• By invoke type: Pull Request=10, Post Merge Action=1, Timer=1

• Pattern: stable

• Monthly unique build counts:

  Month	Unique builds
  2025-07	1
  2025-08	2
  2025-09	0
  2025-10	0
  2025-11	0
  2025-12	2
  2026-01	0
  2026-02	1
  2026-03	0
  2026-04	1
  2026-05	3
  2026-06	2

• Local reproduction: Did not reproduce

  • Command: :qa:mixed-cluster:v2.19.6#mixedClusterTest --tests ... -Dtests.method=...
  • Seed: 9CA5C4992680F376
  • Duration: 159s, gradle exit code 0
  • Notes: BWC mixed-cluster test against v2.19.6 passed in 2m38s.

6. org.opensearch.repositories.s3.S3BlobContainerRetriesTests.classMethod

• Recent failing build: gradle-check #80309 (D24BC247F931822D)

• Original error: java.lang.Exception: Suite timeout exceeded (>= 1200000 msec).

• First seen failing (12M window): 2025-06-21

• Most recent failure: 2026-06-13

• Total unique builds affected (12M): 8

• Committed-code-only unique builds (12M): 1

• By invoke type: Pull Request=8, Timer=4, Post Merge Action=1

• Pattern: stable

• Monthly unique build counts:

  Month	Unique builds
  2025-06	1
  2025-07	2
  2025-08	0
  2025-09	2
  2025-10	0
  2025-11	0
  2025-12	0
  2026-01	0
  2026-02	0
  2026-03	0
  2026-04	1
  2026-05	1
  2026-06	1

• Local reproduction: Did not reproduce

  • Command: full class :plugins:repository-s3:test
  • Seed: D24BC247F931822D
  • Duration: 16s, gradle exit code 0
  • Notes: Full class run completed in 16s with no failures; suite timeout was not reproduced.

7. org.opensearch.arrow.flight.bootstrap.tls.DefaultSslContextProviderFlightIT.testHotReloadWithNewInstance

• Recent failing build: gradle-check #80309 (D24BC247F931822D:706E6763BF8C2A79)

• Original error: org.junit.ComparisonFailure: expected:<cert[2].example.com> but was:<cert[1].example.com>

• First seen failing (12M window): 2026-03-11

• Most recent failure: 2026-06-13

• Total unique builds affected (12M): 7

• Committed-code-only unique builds (12M): 3

• By invoke type: Pull Request=4, Timer=2, Post Merge Action=1

• Pattern: newly observed

• Monthly unique build counts:

  Month	Unique builds
  2026-03	1
  2026-04	0
  2026-05	4
  2026-06	2

• Local reproduction: Did not reproduce

  • Command: :plugins:arrow-flight-rpc:internalClusterTest --tests <method>
  • Seed: D24BC247F931822D:706E6763BF8C2A79
  • Duration: 19s, gradle exit code 0
  • Notes: Single method passed. Timing-sensitive cert reload assertion (cert2 vs cert1).

8. org.opensearch.repositories.s3.S3BlobContainerRetriesTests.testWriteLargeBlob

• Recent failing build: gradle-check #80309 (D24BC247F931822D)

• Original error: java.lang.Exception: Test abandoned because suite timeout was reached.

• First seen failing (12M window): 2025-08-18

• Most recent failure: 2026-06-13

• Total unique builds affected (12M): 2

• Committed-code-only unique builds (12M): 1

• By invoke type: Post Merge Action=1, Pull Request=1

• Pattern: stable

• Monthly unique build counts:

  Month	Unique builds
  2025-08	1
  2025-09	0
  2025-10	0
  2025-11	0
  2025-12	0
  2026-01	0
  2026-02	0
  2026-03	0
  2026-04	0
  2026-05	0
  2026-06	1

• Local reproduction: Did not reproduce

  • Command: :plugins:repository-s3:test --tests <method>
  • Seed: D24BC247F931822D
  • Duration: 16s, gradle exit code 0
  • Notes: Original failure was "Test abandoned because suite timeout was reached" — not a real per-test assertion failure. Single test passed quickly.

9. org.opensearch.indices.replication.RemoteStoreReplicationSourceTests.testGetSegmentFiles

• Recent failing build: gradle-check #80281 (4919CF5F816120E2:DD460E332A3789F0)

• Original error: java.util.concurrent.ExecutionException: org.apache.lucene.index.CorruptIndexException: misplaced codec footer (file truncated?): length=0 but footerLength==16 (resource=metadata__9223372036854775772_

• First seen failing (12M window): 2026-04-03

• Most recent failure: 2026-06-12

• Total unique builds affected (12M): 2

• Committed-code-only unique builds (12M): 2

• By invoke type: Timer=2

• Pattern: newly observed

• Monthly unique build counts:

  Month	Unique builds
  2026-04	1
  2026-05	0
  2026-06	1

• Local reproduction: Did not reproduce

  • Command: :server:test --tests <method>
  • Seed: 4919CF5F816120E2:DD460E332A3789F0
  • Duration: 23s, gradle exit code 0
  • Notes: Single method passed in 23s.

Trend observations

• Several of the highest-volume tests show a step-up starting around April 2026, consistent with the documented gradle-check runner change from m5.8xlarge to the faster m7a.8xlarge. ConcurrentSeqNoVersioningIT.testSeqNoCASLinearizability jumps from 3 builds in 2026-03 to 35 in 2026-04. WarmIndexSegmentReplicationIT.testShardPathDeletionWhenWarmIndexRelocate jumps from 0 in 2026-02/03 to 7 in 2026-04. This is a recurring pattern of CPU-speed amplification of latent timing bugs, not necessarily a regression in those tests' code.
• RestoreShallowSnapshotV2IT is the most consistently broken class in this batch: the suite-level timeout reproduces deterministically on this machine in 20m12s with seed D24BC247F931822D. The cumulative runtime of the class is exceeding the 1200000 ms suite limit, even though no individual test method appears stuck.
• S3BlobContainerRetriesTests suite timeout is environmental — it did not reproduce locally with the same seed (full class ran in 16s). Likely a slow CI VM or a slow retry-window test that occasionally tips over the 20m suite limit.
• MixedClusterClientYamlTestSuiteIT.test {…/Dedup noise} is a YAML-driven significant-text-aggregation test against a mixed-version cluster; each individual occurrence is rare but the test reappears intermittently across many BWC version combinations.

Caveats

• A negative reproduction result here means "did not reproduce in 1 attempt with the given seed on a single machine." It is not evidence that the seed is non-deterministic; the test would need to be run many times to claim that.
• Test-method seeds without reliable reproducibility — like ConcurrentSeqNoVersioningIT.testSeqNoCASLinearizability — depend on thread scheduling, network timing, GC pauses, and other factors RandomizedRunner cannot control. Failure to reproduce here is expected.
• This report intentionally does not attempt to root-cause any failure. It exists to highlight which tests need investigation, in priority order.