Enable PHP FFE evaluation metric system tests

[leoromanovsky]

Motivation

PHP FFE evaluation metrics need system-test coverage before DataDog/dd-trace-php#3911 can be treated as review-ready. The PHP runtime now records feature_flag.evaluations through native/libdatadog-sidecar code, so we need the shared FFE metric tests enabled against a locally built PHP tracer artifact rather than relying only on PHPT coverage.

Design doc: https://docs.google.com/document/d/1NvMfTpZWLBlFmEFNjdnlMyeVpy5l7KD8qujGFco6w2w/edit?tab=t.0

Changes

This PR enables tests/ffe/test_flag_eval_metrics.py for PHP at v1.21.0-dev in the manifest only. It intentionally does not enable exposure tests; exposure validation is tracked separately in #7031 so each PHP milestone can be reviewed independently.

The PR is stacked on #7003, which adds the PHP FFE scaffold and evaluation-test enablement. The PHP behavior being validated here is implemented in DataDog/dd-trace-php#3911 and the sidecar metric delivery support is implemented in DataDog/libdatadog#2052.

Decisions

The scope is deliberately one manifest activation. The shared metric tests cover the expected feature_flag.evaluations behavior, including successful evaluations, missing flags, malformed/empty RC handling, disabled flags, type mismatches, targeting keys, and allocation metadata.

The local proof uses a matched PHP 8.2 NTS artifact and the php-fpm-8.2 weblog, which matches the PHP 8.2 target Bob recommended for FFE system-test iteration.

Related PRs

• PHP runtime evaluation base: feat(ffe): add runtime-backed PHP feature flag evaluation dd-trace-php#3906
• PHP metric implementation: Add FFE evaluation metrics dd-trace-php#3911
• System-tests PHP FFE scaffold/evaluations base: Enable PHP FFE evaluation system tests #7003
• Sidecar metric delivery: feat(sidecar): forward FFE evaluation metrics to OTLP intake libdatadog#2052

Validation

Local behavior validation before this conflict-resolution rebase (previous system-tests PR head a412f4f6), dd-trace-php metric branch now pushed as dc19ce479, and libdatadog submodule 96d9a7bae:

cd /Users/leo.romanovsky/go/src/github.com/DataDog/dd-trace-php-ffe-metrics-restack
./tooling/bin/build-debug-artifact gnu-aarch64-8.2-nts \
  /Users/leo.romanovsky/go/src/github.com/DataDog/system-tests-pr7033/binaries

cd /Users/leo.romanovsky/go/src/github.com/DataDog/system-tests-pr7033
DOCKER_CONFIG=/tmp/system-tests-docker-config-nocredsstore \
SYSTEM_TEST_BUILD_TIMEOUT=1800 \
./build.sh --library php --weblog-variant php-fpm-8.2

DOCKER_CONFIG=/tmp/system-tests-docker-config-nocredsstore \
TEST_LIBRARY=php \
WEBLOG_VARIANT=php-fpm-8.2 \
./run.sh +l php FEATURE_FLAGGING_AND_EXPERIMENTATION tests/ffe/test_flag_eval_metrics.py

Result: 17 passed in 81.28s.

Post-rebase verification: branch head is now fbc98222a, stacked on the updated parent, enabling tests/ffe/test_flag_eval_metrics.py at v1.21.0-dev to match #7003. PATH=/opt/homebrew/opt/coreutils/libexec/gnubin:$PATH ./format.sh passed. The rebase only changed the manifest activation floor; the test behavior and local PHP artifact path above did not change.

MacOS arm64 local-run note: the published PHP 8.2 base image inspected locally was amd64-only, so the local validation reused the locally built arm64 datadog/system-tests:php-fpm-8.2.base-v1. This PR remains only the manifest activation.

[github-actions]

CODEOWNERS have been resolved as:

manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php

[datadog-prod-us1-5]

 

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 26 Pipeline jobs failed

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.1-zts 1     

🔧 Fix in code (Fix with Cursor).

17 failed tests. Assertion errors: Expected metric for various flags found none, indicating possible issues with metric generation.

🧪 1 Test failed

tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Config_Exists_Flag_Missing.test_ffe_eval_config_exists_flag_missing[apache-mod-7.1-zts] from system_tests_suite   (Fix with Cursor)

AssertionError: Expected metric for non-existent flag 'non-existent-eval-metric-flag', found none. All: []
assert 0 > 0
 +  where 0 = len([])

self = <tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Config_Exists_Flag_Missing object at 0x7f470ad89e50>

    def test_ffe_eval_config_exists_flag_missing(self):
        """Test that missing flag (with config loaded) produces error.type=flag_not_found."""
        assert self.r.status_code == 200, f"Flag evaluation request failed: {self.r.text}"
    
...

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.2 1     

🔧 Fix in code (Fix with Cursor).

17 failed tests due to assertion errors: no feature_flag.evaluations metrics found for multiple flags.

🧪 1 Test failed

tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Config_Exists_Flag_Missing.test_ffe_eval_config_exists_flag_missing[apache-mod-7.2] from system_tests_suite   (Fix with Cursor)

AssertionError: Expected metric for non-existent flag 'non-existent-eval-metric-flag', found none. All: []
assert 0 > 0
 +  where 0 = len([])

self = <tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Config_Exists_Flag_Missing object at 0x7f6d8044bd10>

    def test_ffe_eval_config_exists_flag_missing(self):
        """Test that missing flag (with config loaded) produces error.type=flag_not_found."""
        assert self.r.status_code == 200, f"Flag evaluation request failed: {self.r.text}"
    
...

Testing the test | System Tests (php, dev) / End-to-end #1 / apache-mod-7.3 1     

🔧 Fix in code (Fix with Cursor).

17 failed tests due to assertion errors: expected at least one feature_flag.evaluations metric for various flags but found none.

🧪 1 Test failed

tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Config_Exists_Flag_Missing.test_ffe_eval_config_exists_flag_missing[apache-mod-7.3] from system_tests_suite   (Fix with Cursor)

AssertionError: Expected metric for non-existent flag 'non-existent-eval-metric-flag', found none. All: []
assert 0 > 0
 +  where 0 = len([])

self = <tests.ffe.test_flag_eval_metrics.Test_FFE_Eval_Config_Exists_Flag_Missing object at 0x7fee0cca4500>

    def test_ffe_eval_config_exists_flag_missing(self):
        """Test that missing flag (with config loaded) produces error.type=flag_not_found."""
        assert self.r.status_code == 200, f"Flag evaluation request failed: {self.r.text}"
    
...

View all 26 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: fbc9822 | Docs | Datadog PR Page | Give us feedback!}