ci: add Breeze grading workflow - #1424
Merged
Merged
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| env: | ||
| QDC_API_KEY: ${{ secrets.QDC_API_KEY }} | ||
| steps: | ||
| - uses: actions/checkout@v7 |
| env: | ||
| QDC_API_KEY: ${{ secrets.QDC_API_KEY }} | ||
| steps: | ||
| - uses: actions/checkout@v7 |
| --out "${PAYLOAD_FILE}" | ||
|
|
||
| # Downloaded by the agent via a bare `gh run download` -- see issue body. | ||
| - uses: actions/upload-artifact@v7 |
| matrix: ${{ steps.set.outputs.matrix }} | ||
| devices: ${{ steps.set.outputs.devices }} | ||
| steps: | ||
| - uses: actions/checkout@v7 |
| timeout-minutes: 45 | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
| - uses: actions/setup-python@v7 |
| "$THINK" \ | ||
| --prompt-limit "${PROMPT_LIMIT:-0}" \ | ||
| --items-out "$DEVICE-$MODEL_NAME.json" | ||
| - uses: actions/upload-artifact@v7 |
| matrix: ${{ steps.set.outputs.matrix }} | ||
| devices: ${{ steps.set.outputs.devices }} | ||
| steps: | ||
| - uses: actions/checkout@v7 |
| env: | ||
| QDC_API_KEY: ${{ secrets.QDC_API_KEY }} | ||
| steps: | ||
| - uses: actions/checkout@v7 |
| SC*|CRD*|X*) echo "name=sdk-windows-arm64" ;; | ||
| *) echo "name=sdk-linux-arm64" ;; | ||
| esac >> "$GITHUB_OUTPUT" | ||
| - uses: actions/download-artifact@v8 |
| QDC_API_KEY: ${{ secrets.QDC_API_KEY }} | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
| - uses: actions/setup-python@v7 |
| env: | ||
| QDC_API_KEY: ${{ secrets.QDC_API_KEY }} | ||
| steps: | ||
| - uses: actions/checkout@v7 |
| --in-dir items \ | ||
| --grades-in grades.md | ||
|
|
||
| - uses: actions/upload-artifact@v7 |
| if-no-files-found: error | ||
| retention-days: 7 | ||
| # Not part of what the issue tells the grader to download. | ||
| - uses: actions/upload-artifact@v7 |
RemiliaForever (RemiliaForever)
force-pushed
the
ci/breeze-grade
branch
from
September 8, 2026 05:57
96cfa0b to
ef77971
Compare
| - uses: actions/setup-python@v7 | ||
| with: | ||
| python-version: "3.11" | ||
| - uses: actions/download-artifact@v8 |
RemiliaForever (RemiliaForever)
force-pushed
the
ci/breeze-grade
branch
2 times, most recently
from
September 8, 2026 07:39
ce265ae to
a07cefd
Compare
RemiliaForever (RemiliaForever)
marked this pull request as ready for review
September 8, 2026 11:02
Unifies run_qdc_jobs.py's CLI under --mode (bench, bench_aggregate, accuracy, accuracy_payload, accuracy_report) instead of a single --accuracy flag, and adds the accuracy engine: parse a device's geniex-bench log into prompt/response items (one cell per compute unit, so a multi-device run keeps every cell instead of only the first), anonymize and merge every cell across the device x model matrix into one grading payload plus a human-readable item-to-cell map, parse the grader's reply back into per-item ratings, and render two summary tables (score bands, deduction categories) per cell. bench.yml follows the --cells-out/--render-dir -> --out/--in-dir rename so the existing bench pipeline keeps working under the new flag names. Signed-off-by: RemiliaForever <remilia@koumakan.cc>
Each device launcher script gains an accuracy path alongside the existing bench sweep: one pass over the chat-templated prompt file per cell, with the [gen ] lines in the resulting log standing in for the timings bench normally reports. Windows' Start-Transcript writes UTF-16 (handled on the host side); Android has no entry script to substitute a mode into, so params.json carries it and pytest skips whichever of test_bench/test_accuracy the mode did not ask for. Signed-off-by: RemiliaForever <remilia@koumakan.cc>
grade-rules.md tells Breeze how to score quantization damage (0-10, weighted heavily toward decoding failures over ordinary weaknesses) and frames the response text between it grades as untrusted data, not instructions, since it's raw model output graded by an agent with tool access. Signed-off-by: RemiliaForever <remilia@koumakan.cc>
Dispatches geniex-bench --accuracy across a device x model matrix, merges the generated text into an anonymized payload, round-trips it through a GitHub issue to the internal Breeze grader in qcom-ai-hub/geniex, and posts the two aggregate tables plus an item-map artifact back to the run summary. No workflow-level concurrency guard: github.run_id already makes the issue title, QDC job names, and every artifact name unique per run, and QDC job submission already retries through pending-job quota contention, so concurrent runs don't collide anywhere a guard could help. _build-sdk.yml gains the optional HEXAGON_HTP_CERT_PFX secret input this pipeline's build-sdk job passes through. Signed-off-by: RemiliaForever <remilia@koumakan.cc>
Signed-off-by: RemiliaForever <remilia@koumakan.cc>
…eport A generate job that produces zero items (e.g. a device error) used to just vanish from the score/deduction tables. render_grades now takes the full device x model matrix's base cell names and adds a dash row for any not matched by an actual item, so a failed leg shows up as missing instead of being silently dropped. Signed-off-by: RemiliaForever <remilia@koumakan.cc>
The missing-generate-jobs check already computes devices x models; reuse
it to build the {model}-{device} cell list run_qdc_jobs.py needs to
render dash rows for whichever job(s) came back empty.
Signed-off-by: RemiliaForever <remilia@koumakan.cc>
RemiliaForever (RemiliaForever)
force-pushed
the
ci/breeze-grade
branch
from
September 8, 2026 14:32
a07cefd to
7f393e3
Compare
Mengsheng Wu (mengshengwu)
approved these changes
Sep 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs.ymlharness for pre-merge workflow testing; remove it before merge.Test plan
git diff --check.