-
Notifications
You must be signed in to change notification settings - Fork 4.3k
Add agent guidance update skills #32366
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Edward Chen (edgchen1)
wants to merge
16
commits into
main
Choose a base branch
from
edgchen1/agent_guidance_update_skills
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
16 commits
Select commit
Hold shift + click to select a range
493f58d
Add review-driven agent guidance collection skill
edgchen1 a044563
Add agent guidance audit skill
edgchen1 86b780a
Document parallel guidance collection
edgchen1 77edb89
Simplify agent guidance audit marker
edgchen1 28bf601
Merge branch 'main' of github.com:microsoft/onnxruntime into edgchen1…
edgchen1 28f1d3d
Fix collection marker template
edgchen1 a33d102
Clarify review guidance collection decisions
edgchen1 f9ce7d0
Simplify guidance workflow baselines
edgchen1 c91b995
Harden guidance collection metadata
edgchen1 7330d22
lint
edgchen1 dc2fe6e
Run agent skill script tests in CI
edgchen1 209a9ef
lint
edgchen1 891657a
Fail skill tests on empty discovery
edgchen1 fe81b03
Clarify guidance maintenance skill inputs
edgchen1 d19f5a6
remove try/except
edgchen1 81e85c7
Address guidance collection review feedback
edgchen1 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,198 @@ | ||
| --- | ||
| name: audit-agent-guidance | ||
| description: "Audit ONNX Runtime agent guidance for conflicts, stale claims, ineffective scope, duplication, misplaced detail, and opportunities for mechanical enforcement; open a PR that refines, consolidates, relocates, or retires guidance with one logical change per commit." | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why only one change per commit? |
||
| --- | ||
|
|
||
| # Audit ONNX Runtime Agent Guidance | ||
|
|
||
| Use this skill for periodic maintenance of the repository's agent-guidance corpus. Unlike | ||
| `collect-agent-guidance-from-reviews`, which starts from PR feedback, this skill starts from committed guidance and asks | ||
| whether each unit is still correct, well-scoped, discoverable, non-duplicative, and worth its context cost. | ||
|
|
||
| Follow the lifecycle and guidance-layer rules in | ||
| [`docs/Agent_Coding_Guidance.md`](../../../docs/Agent_Coding_Guidance.md). | ||
|
|
||
| ## Safety and Authority | ||
|
|
||
| - Validate guidance claims against the current source code. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: if this applies to instructions and skills what are the 'claims'? |
||
| - Do not remove guidance merely because it has not been cited recently. | ||
| - Do not preserve guidance merely because it is old or was written by an expert. | ||
| - Never weaken a security, ABI, memory-safety, correctness, or compatibility invariant without concrete evidence and | ||
| appropriate owner review. | ||
| - Limit the audit targets to committed repository content, while consulting external references that the guidance | ||
| depends on when necessary. | ||
|
|
||
| ## Scope | ||
|
|
||
| Inventory all agent guidance: | ||
|
|
||
| - `AGENTS.md`; | ||
| - `.github/copilot-instructions.md`; | ||
| - `.github/instructions/**/*.instructions.md`; | ||
| - `.github/skills/**/SKILL.md`; | ||
| - documents explicitly linked from those files as canonical guidance. | ||
|
|
||
| Record the full commit ID of the default-branch state being audited before making changes. Audit only the committed | ||
| guidance and repository state at that snapshot; do not account for changes proposed in open PRs. | ||
|
|
||
| ## Audit Dimensions | ||
|
|
||
| Evaluate each guidance unit against all applicable dimensions. | ||
|
|
||
| ### Technical validity | ||
|
|
||
| - Do referenced paths, symbols, commands, flags, APIs, and tests still exist? | ||
| - Does current implementation behavior still support the stated rationale and correction? | ||
| - Has an architectural change invalidated the guidance? | ||
| - Are version-sensitive statements clearly bounded and current? | ||
|
|
||
| ### Scope and loading | ||
|
|
||
| - Is repository-wide guidance truly universal? | ||
| - Does each `applyTo` pattern cover all intended files without loading for unrelated work? | ||
| - Is a skill description specific enough to trigger for the relevant task? | ||
| - Is detailed subsystem knowledge misplaced in `AGENTS.md` or a broad instruction file? | ||
| - Would moving guidance improve discoverability without duplicating it? | ||
|
|
||
| ### Actionability and discoverability | ||
|
|
||
| - Does the guidance state the triggering condition, failure mode, and required correction? | ||
| - Is the `applyTo` scope or skill description sufficient for an agent to load it for the relevant work? | ||
| - Does the guidance distinguish important exceptions and adjacent cases? | ||
| - Is essential context buried in narrative that an agent is unlikely to apply? | ||
|
|
||
| ### Duplication and conflict | ||
|
|
||
| - Find semantically overlapping guidance, not only repeated wording. | ||
| - Check whether two instructions prescribe incompatible behavior under an overlapping scope. | ||
| - Prefer the narrower, more technically precise invariant. | ||
| - Preserve useful detail by merging into one canonical location and replacing other copies with links when needed. | ||
|
|
||
| ### Enforceability | ||
|
|
||
| - Identify guidance that could be replaced by tests, linters, type-system constraints, safer APIs, schema validation, | ||
| or CI checks. | ||
| - Once mechanical enforcement is established and discoverable, remove redundant prose guidance. | ||
|
|
||
| ### Context value | ||
|
|
||
| - Consolidate historical narrative that does not change agent behavior. | ||
| - Keep rationale, constraints, and non-obvious failure modes that are necessary for correct application. | ||
|
|
||
| ## Audit Outcomes | ||
|
|
||
| Classify each guidance unit as one of: | ||
|
|
||
| | Outcome | Use when | | ||
| |---|---| | ||
| | Retain | The guidance remains correct, useful, and appropriately scoped. | | ||
| | Revise | The guidance needs correction, clarification, rescoping, relocation, consolidation, or separation. | | ||
| | Retire | Guidance is obsolete or fully superseded by maintained mechanical enforcement. | | ||
|
|
||
| Retirement is a normal lifecycle operation, but every retirement requires evidence and a documented successor when one | ||
| exists. | ||
|
|
||
| ## Subagent Delegation | ||
|
|
||
| If detailed validation will not fit reliably in one context, partition the guidance corpus into coherent, | ||
| non-overlapping groups and delegate those groups to subagents using the same audit dimensions. Have subagents return | ||
| findings without editing files. The coordinating agent remains responsible for cross-corpus conflict and duplication | ||
| analysis, final outcomes, and all edits. | ||
|
|
||
| ## Workflow | ||
|
|
||
| ### 1. Build the inventory | ||
|
|
||
| For each guidance unit, record: | ||
|
|
||
| - file and durable heading; | ||
| - guidance layer and effective scope; | ||
| - invariant and rationale; | ||
| - referenced paths, symbols, tools, and documents; | ||
| - related guidance units; | ||
| - relevant origin or refinement PRs available from Git history. | ||
|
|
||
| Use the path and heading as the stable identity defined in | ||
| [`docs/Agent_Coding_Guidance.md`](../../../docs/Agent_Coding_Guidance.md). | ||
|
|
||
| ### 2. Validate current claims | ||
|
|
||
| Read the relevant source and documentation. Use code navigation and history to verify behavioral claims and determine | ||
| whether the guidance remains current. Do not infer technical validity from wording alone. | ||
|
|
||
| ### 3. Inspect provenance and enforcement | ||
|
|
||
| Use the guidance text, its visible source links, current tests and tooling, and Git history to determine: | ||
|
|
||
| - why the guidance was introduced and whether that rationale still applies; | ||
| - whether later repository changes refined or contradicted it; | ||
| - whether mechanical enforcement now makes some or all of the prose redundant; | ||
| - whether a source link or replacement reference is missing or broken. | ||
|
|
||
| Do not broadly rescan PR review comments in this workflow. Review-derived additions and recurrence evidence belong in | ||
| `collect-agent-guidance-from-reviews`. | ||
|
|
||
| ### 4. Design the smallest coherent changes | ||
|
|
||
| For each proposed revision: | ||
|
|
||
| - identify the evidence; | ||
| - state why the current guidance is deficient; | ||
| - select the narrowest canonical destination; | ||
| - check for downstream links or references that must be updated; | ||
| - preserve technically useful rationale and examples; | ||
| - avoid unrelated rewriting or stylistic cleanup. | ||
|
|
||
| ### 5. Prepare independent commits | ||
|
|
||
| Create a branch such as `copilot/audit-agent-guidance-YYYY-MM-DD`. Use one commit per logical guidance change. | ||
| Several files may belong in one commit when moving or deduplicating one invariant. | ||
|
|
||
| Each commit message must identify: | ||
|
|
||
| - the guidance unit being changed; | ||
| - the audit outcome and change; | ||
| - the evidence or current-code basis; | ||
| - source PRs or review comments when relevant. | ||
|
|
||
| ### 6. Validate the proposal | ||
|
|
||
| Before opening the PR: | ||
|
|
||
| - verify every technical claim against the current default branch; | ||
| - ensure moves and merges leave one canonical source without broken links; | ||
| - ensure narrowed scopes still load for every intended path; | ||
| - ensure broadened scopes do not impose subsystem-specific rules on unrelated work; | ||
| - ensure retired guidance has concrete evidence and names its replacement when applicable; | ||
| - ensure each commit corresponds to one row in the proposed-changes table; | ||
| - request appropriate subsystem owners for changes to specialized guidance. | ||
|
|
||
| Open no PR when the audit finds no warranted changes. Report the audited scope and validation performed instead. | ||
|
|
||
| ### 7. Open the PR | ||
|
|
||
| Use this structure: | ||
|
|
||
| ```markdown | ||
| ## Audit scope | ||
|
|
||
| <guidance layers, paths, or subsystems examined> | ||
|
|
||
| ## Snapshot | ||
|
|
||
| Audited commit: `<full commit ID>`. | ||
|
|
||
| ## Proposed changes | ||
|
|
||
| | Commit | Guidance unit | Outcome | Evidence | Replacement or destination | | ||
| |---|---|---|---|---| | ||
|
|
||
| ## Follow-up enforcement work | ||
|
|
||
| | Guidance unit | Recommended test/tooling change | | ||
| |---|---| | ||
|
|
||
| ``` | ||
|
|
||
| Ready-for-review status means the audit proposal is complete enough for maintainers to evaluate. It does not mean the | ||
| proposed guidance changes have been accepted; normal PR approval and merge determine acceptance. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be great to see what is produced by running this skill for changes in say the last month as a draft PR.