OCPMCP-290: add anthropic/gemini vertex jobs - #81118
Conversation
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
|
Skipping CI for Draft Pull Request. |
WalkthroughThe CI config adds two optional MCP checker evaluation steps that mount ChangesOpenShift MCP eval CI steps
Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 14 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (14 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/config/openshift/openshift-mcp-server/openshift-openshift-mcp-server-main.yaml`:
- Around line 67-74: The eval job is masking real failures by forcing mcpchecker
to succeed with “|| true”, so update the job to preserve the actual exit status
while keeping cleanup separate. In the affected check steps for mcpchecker,
replace the inline success override with a trap-based cleanup that always runs
make stop-server, and let mcpchecker return its real status so the optional jobs
still report failures correctly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 811f0b2c-ad67-458f-9213-c8216f02c0e5
⛔ Files ignored due to path filters (1)
ci-operator/jobs/openshift/openshift-mcp-server/openshift-openshift-mcp-server-main-presubmits.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (1)
ci-operator/config/openshift/openshift-mcp-server/openshift-openshift-mcp-server-main.yaml
|
/pj-rehearse |
|
@dlom: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse |
|
@grokspawn: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/hold |
openshift-ci
Bot
added
the
do-not-merge/hold
968d084 to
f08c1ea
Compare
|
/pj-rehearse |
|
@grokspawn: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
Caution Review failedAn error occurred during the review process. Please try again later. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
/pj-rehearse |
|
@grokspawn: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/hold cancel |
openshift-ci
Bot
removed
the
do-not-merge/hold
|
containers/kubernetes-mcp-server#1241 merged, but needs to be downstreamed before this can be re-evaluated |
|
This appears sound, but cannot merge w/o the extra rehearsal jobs, so I'll switch this from draft but with a hold until those pass. |
openshift-ci
Bot
removed
the
do-not-merge/work-in-progress
|
/hold |
openshift-ci
Bot
added
the
do-not-merge/hold
|
/pj-rehearse |
|
@grokspawn: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
wking
left a comment
wking
left a comment
There was a problem hiding this comment.
There's a new OWNERS file for all the new steps, and it doesn't have me in it, so I'm happy to delegate to the new owners to be responsible for the new step, without having to form my own opinions about what's inside the new steps. Adding a hold in case you want to address this:
/hold in case you want OWNERS symlinks
But feel free to lift the hold without making changes if you like, no worries from me.
openshift-ci
Bot
added
the
do-not-merge/hold
openshift-ci
Bot
added
the
approved
openshift-merge-bot
Bot
removed
the
rehearsals-ack
|
/pj-rehearse auto-ack |
|
@grokspawn: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
Signed-off-by: grokspawn <jordan@nimblewidget.com>
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
/pj-rehearse |
|
@grokspawn: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Cali0707, grokspawn, wking The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/hold cancel |
openshift-ci
Bot
removed
the
do-not-merge/hold
|
@grokspawn: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/pj-rehearse ack |
|
@grokspawn: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-merge-bot
Bot
added
the
rehearsals-ack
![openshift-prow[bot]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
* add anthropic/gemini vertex jobs * experimenting with semi-hermetic build env Signed-off-by: grokspawn <jordan@nimblewidget.com> * switch to step registry and parameterized workflows Signed-off-by: grokspawn <jordan@nimblewidget.com> --------- Signed-off-by: grokspawn <jordan@nimblewidget.com>
* add anthropic/gemini vertex jobs * experimenting with semi-hermetic build env Signed-off-by: grokspawn <jordan@nimblewidget.com> * switch to step registry and parameterized workflows Signed-off-by: grokspawn <jordan@nimblewidget.com> --------- Signed-off-by: grokspawn <jordan@nimblewidget.com>
6 participants
Builds off #78018, but switches to a multi-step test against OCP clusters as a general framework for executing MCP evals against arbitrary models with flexible evaluation criteria. For this initial work, the evaluation success gate is not set since not all evals are currently configured to be able to be successfully executed in a downstream environment, but this is a separate issue than "can we automate it".
Summary by CodeRabbit
This PR updates OpenShift CI configuration for the MCP server to add two optional Vertex-based evaluation steps (Google Gemini and Anthropic Claude). In practical terms, it switches to a multi-step inline
mcpcheckerworkflow so the job can access mounted credentials during the test run.What changed
ci-operator/config/openshift/openshift-mcp-server/openshift-openshift-mcp-server-main.yamlto add two new optional MCP checker evaluation steps:mcpchecker-eval-googlemcpchecker-eval-anthropicocp-mcpcredential (fromtest-credentials) at/var/run/ocp-mcpset -euo pipefail)GOOGLE_CLOUD_LOCATION=us-east1andGEMINI_USE_VERTEX=1GOOGLE_CLOUD_LOCATION=us-east1andANTHROPIC_USE_VERTEX=1/var/run/ocp-mcp/service-account.jsonGOOGLE_CLOUD_PROJECTby parsing the mounted service account JSON via an inlinepython3 -ccommandGOFLAGS='-mod=readonly' make buildGOFLAGS='' make mcpcheckermake run-server TOOLSETS=core,config)trap 'make stop-server || true' EXIT)_output/tools/bin/mcpchecker checkin JSON mode (--output json) with--label-selector suite=coregrace_period: 1m0sand resources (cpu: 500m,memory: 200Mi)Eval suites used
mcpchecker-eval-google:evals/core-eval-testing/builtin-google/eval-core.yamlmcpchecker-eval-anthropic:evals/core-eval-testing/builtin-anthropic/eval-core.yamlCI impact
optional: true, so they can be exercised without necessarily blocking the overall MCP server CI run if they fail.