OCPMCP-108: weekly mcpchecker eval periodic for openshift-mcp-server - #81947
OCPMCP-108: weekly mcpchecker eval periodic for openshift-mcp-server#81947cajieh wants to merge 3 commits into
Conversation
|
@cajieh: This pull request references OCPMCP-108 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
added
the
jira/valid-reference
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds a weekly CI job that runs the MCP server with ChangesMCP checker evaluation
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant periodic-mcpchecker-eval
participant MCPServer
participant mcpchecker
participant ARTIFACT_DIR
periodic-mcpchecker-eval->>MCPServer: make run-server TOOLSETS=core,config
periodic-mcpchecker-eval->>mcpchecker: check eval-core.yaml with suite=core
mcpchecker-->>periodic-mcpchecker-eval: JSON evaluation results
periodic-mcpchecker-eval->>ARTIFACT_DIR: Copy JSON and write JUnit XML
periodic-mcpchecker-eval->>MCPServer: Stop server on exit
Suggested reviewers: 🚥 Pre-merge checks | ✅ 14 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (14 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
ci-operator/config/openshift/openshift-mcp-server/openshift-openshift-mcp-server-main.yaml (1)
18-18: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winGo patch version is behind latest security release.
Go 1.26.3 go1.26.3 (released 2026-05-07) includes security fixes to the go command, the pack tool, and the html/template, net, net/http, net/http/httputil, net/mail, and syscall packages, but newer patches exist: go1.26.4 (released 2026-06-02) includes security fixes to the crypto/x509, mime, and net/textproto packages... go1.26.5 (released 2026-07-07) includes security fixes to the crypto/tls and os packages. Consider bumping to 1.26.5 to pick up the latest security fixes.
🔧 Proposed fix
- RUN curl -fsSL https://go.dev/dl/go1.26.3.linux-amd64.tar.gz | tar -C /usr/local -xzf - \ + RUN curl -fsSL https://go.dev/dl/go1.26.5.linux-amd64.tar.gz | tar -C /usr/local -xzf - \🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/config/openshift/openshift-mcp-server/openshift-openshift-mcp-server-main.yaml` at line 18, Update the Go toolchain download URL in the Docker build RUN command to use Go 1.26.5 instead of 1.26.3, keeping the existing Linux amd64 archive extraction flow unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In
`@ci-operator/config/openshift/openshift-mcp-server/openshift-openshift-mcp-server-main.yaml`:
- Line 18: Update the Go toolchain download URL in the Docker build RUN command
to use Go 1.26.5 instead of 1.26.3, keeping the existing Linux amd64 archive
extraction flow unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 94040479-07ab-40ea-a1df-9508cd873ac9
⛔ Files ignored due to path filters (1)
ci-operator/jobs/openshift/openshift-mcp-server/openshift-openshift-mcp-server-main-periodics.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (1)
ci-operator/config/openshift/openshift-mcp-server/openshift-openshift-mcp-server-main.yaml
3320f6e to
abd97a3
Compare
|
/retest |
ffdeba9 to
4b4d1b8
Compare
|
/pj-rehearse periodic-ci-openshift-openshift-mcp-server-main-periodic-mcpchecker-eval |
|
@cajieh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
4b4d1b8 to
fdc3fda
Compare
|
/pj-rehearse periodic-ci-openshift-openshift-mcp-server-main-periodic-mcpchecker-eval |
1 similar comment
|
/pj-rehearse periodic-ci-openshift-openshift-mcp-server-main-periodic-mcpchecker-eval |
|
@cajieh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
fdc3fda to
4b19743
Compare
|
/pj-rehearse periodic-ci-openshift-openshift-mcp-server-main-periodic-mcpchecker-eval |
|
@cajieh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse periodic-ci-openshift-openshift-mcp-server-main-periodic-mcpchecker-eval |
|
@cajieh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
4b19743 to
2e59c79
Compare
|
@cajieh: |
2e59c79 to
e350a20
Compare
|
/pj-rehearse periodic-ci-openshift-openshift-mcp-server-main-periodic-mcpchecker-eval |
|
@cajieh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
| GOFLAGS='-mod=readonly' make build | ||
| GOFLAGS='' make mcpchecker | ||
|
|
||
| make run-server TOOLSETS="${TOOLSETS}" |
There was a problem hiding this comment.
Note on why we don't call make run-server here:
make run-server starts the MCP server without --kubeconfig / --cluster-provider. In Prow, pods expose KUBERNETES_SERVICE_*, so the server selects the build-farm in-cluster config and hits RBAC denials against the IPI test cluster.
This step intentionally starts ./openshift-mcp-server with:
unset KUBERNETES_SERVICE_HOST KUBERNETES_SERVICE_PORT--kubeconfig "${KUBECONFIG}"(${SHARED_DIR}/kubeconfigfromipi-aws)--cluster-provider kubeconfig
We still write .mcp-server.pid so make stop-server in the trap works.
Follow-up: Extend make run-server in openshift/openshift-mcp-server to pass --kubeconfig / --cluster-provider, then switch this step back to make run-server TOOLSETS=....
There was a problem hiding this comment.
This addresses the provider determination steps in make run-server. Since there is no current capability to pass a kubeconfig, the election selects the pod-scoped service account (which has no privs).
Yesterday I was playing with doing a drop-in toml for the CI run which works around it, but I think it's far more straightforward to expand the upstream target so that we're not having to generate non-standard config (meaning that there are too many ways we can customize this and easy to lose track of which path does it which way).
I see no reason why we cannot solve this in a follow-up, but I'd want to be very careful not to forget this tech debt.
e350a20 to
ea37ce9
Compare
|
/pj-rehearse periodic-ci-openshift-openshift-mcp-server-main-periodic-mcpchecker-eval |
|
@cajieh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
ea37ce9 to
c871370
Compare
|
/pj-rehearse periodic-ci-openshift-openshift-mcp-server-main-periodic-mcpchecker-eval |
|
@cajieh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
c871370 to
1c0f9ee
Compare
|
/pj-rehearse periodic-ci-openshift-openshift-mcp-server-main-periodic-mcpchecker-eval |
|
@cajieh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
1c0f9ee to
27e43a5
Compare
|
/pj-rehearse periodic-ci-openshift-openshift-mcp-server-main-periodic-mcpchecker-eval |
|
@cajieh: your |
|
/pj-rehearse periodic-ci-openshift-openshift-mcp-server-main-periodic-mcpchecker-eval |
|
@cajieh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@cajieh: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
| TOKEN_PATH="/var/run/ocp-mcp/openai-token" | ||
|
|
||
| $WAS_TRACING && set -x | ||
| # Show mount contents without printing secret values. |
There was a problem hiding this comment.
These logs are intentional. Early rehearses failed at credentials-setup with little signal (missing secret vs unreadable key vs empty file). Happy to trim later once the job is stable.
| # Matches Makefile BINARY_NAME in openshift/openshift-mcp-server | ||
| # (go build -o kubernetes-mcp-server). | ||
| BINARY_NAME="${BINARY_NAME:-kubernetes-mcp-server}" | ||
| if [[ ! -x "./${BINARY_NAME}" ]]; then |
There was a problem hiding this comment.
Also intentional: hit a silent healthz timeout when the binary name was wrong (openshift-mcp-server vs Makefile kubernetes-mcp-server). Checking the binary exists and dumping mcp-server.log on readiness failure makes that class of issue show up immediately in the step log.
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: cajieh, grokspawn The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
we didn't manage to get the Makefile changes merged yesterday, so they would be included in the downstream sync overnight. 😞 |
Thanks for the feedback! I'd prefer to handle this in a follow-up so we can get this merged first. I will open an issue to keep track of it. |
Define the weekly job on the release-5.0 config so Sippy can attach results to an OCP release, not only the main-branch periodic. Co-authored-by: Cursor <cursoragent@cursor.com>
|
New changes are detected. LGTM label has been removed. |
Keep the weekly only on release-5.0 so Sippy can align the job with an OCP release; avoid a duplicate main-branch weekly. Co-authored-by: Cursor <cursoragent@cursor.com>
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
/pj-rehearse periodic-ci-openshift-openshift-mcp-server-main-periodic-mcpchecker-eval |
|
@cajieh: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@cajieh: job(s): periodic-ci-openshift-openshift-mcp-server-main-periodic-mcpchecker-eval either don't exist or were not found to be affected, and cannot be rehearsed |
3 participants
Summary
periodic-mcpchecker-eval, Mon 09:00 UTC) that runs mcpchecker core-suite evals againstopenshift/openshift-mcp-serveron a live OpenShift cluster (ipi-aws/openshift-org-aws), with an 80% verify gate and JSON + JUnit artifacts under${ARTIFACT_DIR}for Spyglass / Sippy (OCPMCP-108 / OCPMCP-308.1).openshift-mcp-server-mcpchecker-eval-model-api(credentials-setup → eval), parameterized viaEVAL_CONFIG/ pass-rate env vars — no inline test script.evals/core-eval-testing/builtin-openai/eval-core.yaml(judge via sharedagent.yamlllmJudge.ref; no deprecatedJUDGE_*env vars).ocp-mcpcredentials fromtest-credentialsat/var/run/ocp-mcpand exportsOPENAI_API_KEYfrom theopenai-tokenkey (Vault:selfservice/ocp-mcp/openai-credentials).junit_mcpchecker.xmlbefore the verify gate so Spyglass/Sippy still get artifacts on gate failure.Test plan
ci-operator-config/generated-config/yamllintpass on this PRopenai-tokenis synced into theocp-mcpCI secret/pj-rehearse periodic-ci-openshift-openshift-mcp-server-main-periodic-mcpchecker-evalafter OpenAI token is available in CI${ARTIFACT_DIR}/mcpchecker-out.jsonand${ARTIFACT_DIR}/junit_mcpchecker.xmlCo-authored by Claude and Cursor.