Enable PHP FFE evaluation system tests by leoromanovsky · Pull Request #7003 · DataDog/system-tests

leoromanovsky · 2026-05-21T22:12:26Z

Motivation

The PHP Feature Flags and Experimentation evaluation runtime has landed in dd-trace-php and is published through the v1.21.0-dev artifact. PHP should run the shared FFE system tests so the runtime path is covered by live FFE_FLAGS Remote Config delivery and by the canonical JSON evaluation fixtures used across libraries.

Reference design doc: https://docs.google.com/document/d/1NvMfTpZWLBlFmEFNjdnlMyeVpy5l7KD8qujGFco6w2w/edit?tab=t.0

Changes

This PR enables the existing end-to-end tests/ffe/test_dynamic_evaluation.py suite for PHP starting at v1.21.0-dev and adds the minimal PHP weblog /ffe endpoint needed by that shared suite. The endpoint calls the DDTrace\FeatureFlags\Client details API, normalizes the request shape used by system-tests, and returns provider-not-ready defaults when the runtime is unavailable.

This PR also enables the parametric canonical JSON fixture evaluation suite for PHP. The PHP parametric server implements /ffe/start and /ffe/evaluate using the same Datadog client API, and OF.7 is covered through test-case-of-7-empty-targeting-key.json in the shared parametrized test.

Decisions

The parametric evaluation test still sets FFE_FLAGS through Remote Config and waits for the test-agent to observe an acknowledged apply state. For the PHP parametric CLI process, that ACK is not enough to make the native evaluator report loaded config: CI on 1d6ac67 showed every dynamic evaluation case still returning PROVIDER_NOT_READY with providerState.hasConfig == false after the retry loop.

To keep the parametric suite deterministic, /ffe/start accepts the canonical UFC fixture payload uniformly. Existing non-PHP parametric servers ignore that optional payload; PHP uses it to seed the native evaluator through DDTrace\Testing\ffe_load_config before constructing the client. The shared test still has no PHP branch, and the explicit readiness check remains so failures show whether the provider is still waiting for config.

The scope is deliberately evaluation-only. Exposure forwarding and feature_flag.evaluations metrics remain disabled here so they can be enabled in separate stacked PRs after their runtime implementations are ready.

Verification

Current branch checks:

$ ./format.sh
All good, the system-tests CI will be happy!

$ TEST_LIBRARY=php ./run.sh PARAMETRIC --skip-parametric-build tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Dynamic_Evaluation --collect-only -q
24 tests collected

The collected set includes test-case-of-7-empty-targeting-key.json, so OF.7 is covered by the generic JSON parametrization.

github-actions · 2026-05-21T22:12:58Z

CODEOWNERS have been resolved as:

utils/build/docker/php/common/ffe.php                                   @DataDog/apm-php @DataDog/system-tests-core
docs/understand/weblogs/end-to-end_weblog.md                            @DataDog/system-tests-core
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
tests/parametric/test_ffe/test_dynamic_evaluation.py                    @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
utils/build/docker/php/common/rewrite-rules.conf                        @DataDog/apm-php @DataDog/system-tests-core
utils/build/docker/php/parametric/server.php                            @DataDog/apm-php @DataDog/system-tests-core
utils/docker_fixtures/_test_clients/_test_client_parametric.py          @DataDog/system-tests-core

datadog-prod-us1-4 · 2026-05-22T19:03:15Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 552010f | Docs | Datadog PR Page | Give us feedback!}

bwoebi · 2026-05-28T23:35:23Z

+    def ffe_start(self, configuration: dict | None = None) -> bool:
        """Initialize the FFE (Feature Flagging & Experimentation) provider."""
-        return self._client.ffe_start()
+        return self._client.ffe_start(configuration)


You don't have to do this, you can also rely on RC polling and do a small loop with a few sleeps until RC data is available?

leoromanovsky · 2026-05-29T13:56:46Z

  tests/parametric/test_dynamic_configuration.py::TestDynamicConfigV2: v2.44.0
  tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Dynamic_Evaluation: '>=3.36.0'  # Modified by easy win activation script
  tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Dynamic_Evaluation::test_ffe_flag_evaluation: missing_feature  # Created by easy win activation script
-  tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Dynamic_Evaluation::test_ffe_of7_empty_targeting_key: missing_feature  # Created by easy win activation script


This was fixed; removing

leoromanovsky · 2026-05-29T13:56:49Z

  tests/parametric/test_dynamic_configuration.py::TestDynamicConfigV1_ServiceTargets::test_not_match_service_target: irrelevant (APMAPI-1003)
  tests/parametric/test_dynamic_configuration.py::TestDynamicConfigV2: v1.31.0
  tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Dynamic_Evaluation: v1.56.0
-  tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Dynamic_Evaluation::test_ffe_of7_empty_targeting_key: bug (FFL-1729)


This was fixed; removing

leoromanovsky · 2026-05-29T13:56:53Z

  tests/parametric/test_dynamic_configuration.py::TestDynamicConfigV1_ServiceTargets::test_not_match_service_target: bug (APMAPI-865)
  tests/parametric/test_dynamic_configuration.py::TestDynamicConfigV2: *ref_4_23_0
  tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Dynamic_Evaluation: *ref_5_75_0
-  tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Dynamic_Evaluation::test_ffe_of7_empty_targeting_key: bug (FFL-1730)


This was fixed; removing

leoromanovsky · 2026-05-29T13:57:10Z

-        # Skip OF.7 (empty targeting key) test for libraries with known bugs
-        # Java: FFL-1729 - OpenFeature Java SDK rejects empty targeting keys
-        # Node.js: FFL-1730 - OpenFeature JS SDK rejects empty targeting keys
-        if test_case_file == "test-case-of-7-empty-targeting-key.json":
-            if context.library.name in ("java", "nodejs"):
-                pytest.skip("OF.7 empty targeting key bug: FFL-1729 (java), FFL-1730 (nodejs)")


This was fixed; removing - works for php as well

leoromanovsky · 2026-05-29T13:57:33Z

-    @parametrize("library_env", [{**DEFAULT_ENVVARS}])
-    def test_ffe_of7_empty_targeting_key(self, test_agent: TestAgentAPI, test_library: APMLibrary) -> None:
-        """OF.7: Empty string is a valid targeting key.


special test not needed; using the centralized fixtures again for all languages

leoromanovsky · 2026-05-29T15:48:20Z

Merge activity

May 29, 3:48 PM UTC: A user started a stack merge that includes this pull request via Graphite.
May 29, 3:48 PM UTC: @leoromanovsky merged this pull request with Graphite.

leoromanovsky mentioned this pull request May 22, 2026

feat(ffe): add runtime-backed PHP feature flag evaluation DataDog/dd-trace-php#3906

Merged

leoromanovsky changed the title ~~Add PHP FFE system-test scaffold~~ Enable PHP FFE parametric system tests May 22, 2026

leoromanovsky force-pushed the leo.romanovsky/pr-g-php-ffe-scaffold branch from 4ca7546 to 984548d Compare May 28, 2026 02:54

leoromanovsky changed the title ~~Enable PHP FFE parametric system tests~~ Enable PHP FFE evaluation system tests May 28, 2026

test(ffe): enable PHP evaluation system tests

224c241

leoromanovsky force-pushed the leo.romanovsky/pr-g-php-ffe-scaffold branch from 984548d to 224c241 Compare May 28, 2026 03:26

leoromanovsky added 4 commits May 28, 2026 11:07

Merge branch 'main' into leo.romanovsky/pr-g-php-ffe-scaffold

8fbb3a5

Fix PHP FFE test activation versions

9bcf43d

Skip stale PHP FFE dev build

e2d4370

Bump PHP FFE tests to 1.21 dev

19b9c41

leoromanovsky marked this pull request as ready for review May 28, 2026 21:09

leoromanovsky requested review from a team as code owners May 28, 2026 21:09

leoromanovsky requested review from sameerank and typotter and removed request for a team May 28, 2026 21:10

sameerank reviewed May 28, 2026

View reviewed changes

Comment thread manifests/php.yml

Comment thread tests/parametric/test_ffe/test_dynamic_evaluation.py Outdated

Comment thread tests/parametric/test_ffe/test_dynamic_evaluation.py Outdated

leoromanovsky added 2 commits May 28, 2026 18:21

Remove PHP-specific FFE test branching

83bc16b

Use OF7 JSON fixture in FFE parametric tests

afd0d5f

leoromanovsky requested review from a team as code owners May 28, 2026 23:30

leoromanovsky requested review from amarziali, daniel-romano-DD and jandro996 May 28, 2026 23:30

bwoebi reviewed May 28, 2026

View reviewed changes

leoromanovsky added 2 commits May 28, 2026 19:42

Merge origin/main into PHP FFE tests

1d6ac67

Restore deterministic PHP FFE fixture startup

552010f

leoromanovsky commented May 29, 2026

View reviewed changes

leoromanovsky requested a review from sameerank May 29, 2026 13:57

sameerank approved these changes May 29, 2026

View reviewed changes

leoromanovsky merged commit 56306b6 into main May 29, 2026
1392 of 1395 checks passed

leoromanovsky deleted the leo.romanovsky/pr-g-php-ffe-scaffold branch May 29, 2026 15:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable PHP FFE evaluation system tests#7003

Enable PHP FFE evaluation system tests#7003
leoromanovsky merged 9 commits into
mainfrom
leo.romanovsky/pr-g-php-ffe-scaffold

leoromanovsky commented May 21, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 21, 2026 •

edited

Loading

Uh oh!

datadog-prod-us1-4 Bot commented May 22, 2026 •

edited by datadog-datadog-prod-us1 Bot

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bwoebi May 28, 2026

Uh oh!

leoromanovsky May 29, 2026

Uh oh!

leoromanovsky May 29, 2026

Uh oh!

leoromanovsky May 29, 2026

Uh oh!

leoromanovsky May 29, 2026

Uh oh!

leoromanovsky May 29, 2026

Uh oh!

leoromanovsky commented May 29, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

leoromanovsky commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Decisions

Verification

Uh oh!

github-actions Bot commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-prod-us1-4 Bot commented May 22, 2026 • edited by datadog-datadog-prod-us1 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bwoebi May 28, 2026

Choose a reason for hiding this comment

Uh oh!

leoromanovsky May 29, 2026

Choose a reason for hiding this comment

Uh oh!

leoromanovsky May 29, 2026

Choose a reason for hiding this comment

Uh oh!

leoromanovsky May 29, 2026

Choose a reason for hiding this comment

Uh oh!

leoromanovsky May 29, 2026

Choose a reason for hiding this comment

Uh oh!

leoromanovsky May 29, 2026

Choose a reason for hiding this comment

Uh oh!

leoromanovsky commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge activity

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

leoromanovsky commented May 21, 2026 •

edited

Loading

github-actions Bot commented May 21, 2026 •

edited

Loading

datadog-prod-us1-4 Bot commented May 22, 2026 •

edited by datadog-datadog-prod-us1 Bot

Loading

leoromanovsky commented May 29, 2026 •

edited

Loading