Create a weekly audit skill that can post an issue with the audit. by gspencergoog · Pull Request #2018 · a2ui-project/a2ui

gspencergoog · 2026-07-16T16:38:23Z

Summary

This PR establishes a suite of automated tools, playbooks, and unit tests to support repository maintenance, focusing on weekly A2UI compliance verification and issue reporting. It introduces automated check and reporting scripts, integrated as specialized agent skills with high test coverage.

Changes

📋 `a2ui-audit` Skill

Added SKILL.md establishing playbooks for multi-language test coverage, dependency management, link checking, formatting, and blueprint compliance checks.
Added gh-reference.md containing read-only commands for using gh to search code, check issues, view PR reviews, and inspect CI workflow run logs safely.
Added create_compliance_report.py to compile the weekly report and post it to GitHub issues with the title "Weekly A2UI Compliance Report".
Added test_create_compliance_report.py providing unit tests for the compliance reporting script.

🔍 `a2ui-blueprint-compliance` Skill

Added check_compliance.py to automate discovering codebase blueprints, parsing pinned commits, and comparing them against module blueprint git logs to generate a Markdown report.
Added test_check_compliance.py providing coverage for check, baseline, missing commit parsing, and error status flows.
Updated SKILL.md instructing agents to run the compliance script first to automate baseline reports.

🛠️ Infrastructure & Integration

Modified link_skills.sh to link the compliance skill.
Added new scripts to .gitignore.
Updated MyPy configuration in pyproject.toml to support type checks for blueprints/ and specify search paths for cross-package imports.

Impact & Risks

None. All scripts, references, and tests are scoped to local developer skills and CI checks; no changes affect core protocol specifications or SDK production code.

Testing

Skill Unit Tests: Verified all skill unit tests pass successfully:
```
python3 scripts/run_skills_tests.py
```
Code Coverage: Verified code coverage exceeds the 90% coverage target using pytest-cov on both scripts:
- check_compliance.py: 92% coverage
- create_compliance_report.py: 94% coverage
Blueprint Validation: Verified all blueprints pass structural/schema validations:
```
python3 blueprints/validate_blueprints.py
```

… SKILL.md

…skill

… script

…hashes safely

…nce_report

…nd tests

…eck_compliance script

…s and check_compliance

…p triage logic in try-catch

…g to improve CI resilience against transient API 503s

josemontespg

Looks good to me. Please also get approval from @jacobsimionato for the blueprint part.

josemontespg · 2026-07-16T23:36:19Z

This is very easily extensible to other audit tasks that we might come up in the future, nice!

josemontespg · 2026-07-16T23:39:09Z

+
+## **Instructions**
+
+1. **Audit Codebase Blueprint Compliance**:


Optional: Probably doesn't matter too much, but each of these steps is well suited to be executed by subagents. We could instruct here to use subagents to delegate each of the tasks, to keep the main agent context window clean, and the subtasks well scoped.

Yeah, I considered that, and maybe we should, but I was worried that it might increase the likelihood of exceeding rate limits, and since this is running offline, speed doesn't really matter.

josemontespg · 2026-07-16T23:41:47Z

@jacobsimionato deferring to your review for these files, as you are the expert in blueprints

jacobsimionato · 2026-07-16T23:24:14Z

+This reference provides information about using the GitHub CLI (`gh`) to perform read-only investigations of repository state, and to manage issues and pull requests.
+
+> [!IMPORTANT]
+> If the GITHUB_TOKEN environment variable has a dummy value in it, to run the `gh` command-line tool, you must prefix it with `env -u GITHUB_TOKEN` to unset the dummy environment variable (e.g., `env -u GITHUB_TOKEN gh auth status`). Don't do this if the token appears to be a valid token.


Why do we need to have environments with a dummy GITHUB_TOKEN? Could we just not have them to avoid this?

Antigravity sets a dummy value for the token internally, so the agent can't run it while that value is set. This works around that.

jacobsimionato · 2026-07-16T23:32:01Z

+- Refer to the [`a2ui-blueprint-compliance`](../../../blueprints/skills/a2ui-blueprint-compliance/SKILL.md) skill for codebase blueprint checking.
+- Refer to the [`a2ui-doc-sync-check`](../a2ui-doc-sync-check/SKILL.md) skill for documentation sync checking.
+- Refer to the [`a2ui-test-quality-check`](../a2ui-test-quality-check/SKILL.md) skill for test quality and assertion checking.
+- Refer to the [`gh-reference`](./references/gh-reference.md) reference for GitHub CLI (`gh`) operations.


Seeing as there is a wrapper script for creating the Github issue anyway, what is gh-reference needed for? Does the agent typically check Github issues to understand project context etc?

Yes, and if you look at the reference, it's all about investigation: it doesn't include reference information about write operations (which it doesn't have access to anyhow). If it needs to fetch issues, PRs, or other information it can use gh to do it. I suppose not strictly required, but it seemed like something that would be useful for collecting information.

jacobsimionato · 2026-07-16T23:42:50Z

When you initially proposed this idea of thinking of blueprints as versioned documents, it struck me as too complex.

But now that I understand your vision for the tooling around this, I really love this model. Every time a module blueprint is updated, we can ensure that each codebase is updated. I hope it works as well in practice as it does in my head!

We'll see... It's all a grand experiment.

jacobsimionato · 2026-07-16T23:46:36Z

+       - Features listed in `implemented_features` that are missing or incomplete in the code.
+       - Implemented features that are missing from the `implemented_features` frontmatter list.
+     - **Implemented Optional Features**: A list of optional features verified as implemented in the code.
+     - **Missing Commits / Spec Diffs**: The list of git commits since the pinned hash.


Potential future idea: Given the model will have already spent a lot of tokens looking through the code at this point, I wonder if it's good value for it to write down some proposals outlining the rough process required to fix discrepancies that are found, e.g. missing features, incorrectly implemented etc.

It's be great to work towards a state where from this report, you're one click away from asking an agent to go and fix each issue. E.g. the report includes proposals with enough context that a human can review and be confident the proposal will send an agent in the right direction.

Yes, that would be a good step. Right now, I want to see if this report is useful, and refine it until it is. I can see the bot also filing draft PRs and posting them here, but that feels like it might be too noisy.

I did consider somehow staging the PRs so that you could just click on a link in the report and have it generate a PR (but not generate the PR automatically until the button is pushed), but I'm not sure how that would be set up in practice, and might be too complex.

# Conflicts: # pyproject.toml

Create a weekly audit skill that can post an issue with the audit.

be148b9

github-project-automation Bot added this to A2UI Jul 16, 2026

github-project-automation Bot moved this to Todo in A2UI Jul 16, 2026

This comment was marked as resolved.

Sign in to view

gspencergoog added 4 commits July 16, 2026 09:41

fix: resolve broken links to blueprint-compliance skill in a2ui-audit…

1a0e9c6

… SKILL.md

test(a2ui-audit): add unit tests for post_compliance_report script

c0cbf2b

docs(gh-cli): fix incorrect commands, subcommands, and flags

523ac0c

Clean up GH reference and make it a reference instead of a skill

85d36c7

gspencergoog force-pushed the a2ui-audit branch from 3665569 to 85d36c7 Compare July 16, 2026 17:07

gspencergoog added 3 commits July 16, 2026 10:29

Merge branch 'main' into a2ui-audit

f7729b9

feat(blueprint): add check_compliance script and tests to compliance …

5b8b72b

…skill

refactor(blueprint): remove issue posting logic from check_compliance…

5043cff

… script

gspencergoog requested review from jacobsimionato and josemontespg July 16, 2026 21:23

This comment was marked as resolved.

Sign in to view

fix(blueprint-compliance): handle missing executable and None commit …

43002cf

…hashes safely

gspencergoog force-pushed the a2ui-audit branch from 700a9d7 to 43002cf Compare July 16, 2026 21:49

Merge branch 'main' into a2ui-audit

2215957

gspencergoog force-pushed the a2ui-audit branch from f315608 to 2215957 Compare July 16, 2026 22:07

gspencergoog added 9 commits July 16, 2026 15:17

refactor(a2ui-audit): rename post_compliance_report to create_complia…

54ccdd0

…nce_report

docs(a2ui-audit): add docstrings to create_compliance_report script a…

58acd00

…nd tests

docs(blueprint-compliance): add docstrings and type annotations to ch…

5bdc421

…eck_compliance script

docs(blueprints): resolve type issues and document validate_blueprint…

4431ed9

…s and check_compliance

Fix title

8026f0c

test(triage): fix mock paginate and comment factory in unit tests

7b234f4

ci(web_ci): fetch full git history in changes job; chore(triage): wra…

83d953f

…p triage logic in try-catch

chore(triage): log triage reconciliation errors instead of re-throwin…

2d1788c

…g to improve CI resilience against transient API 503s

revert: remove CI mitigations for transient GitHub API outage

76f9271

gspencergoog marked this pull request as ready for review July 16, 2026 22:54

Revert CI script changes

171c2f7

josemontespg approved these changes Jul 16, 2026

View reviewed changes

jacobsimionato reviewed Jul 17, 2026

View reviewed changes

gspencergoog and others added 3 commits July 17, 2026 10:29

Merge branch 'main' into a2ui-audit

0c7566c

Merge branch 'main' into a2ui-audit

3303a5d

Merge branch 'main' into a2ui-audit

5c7971e

# Conflicts: # pyproject.toml

gspencergoog enabled auto-merge (squash) July 17, 2026 22:45

gspencergoog merged commit 3708c06 into a2ui-project:main Jul 17, 2026
23 checks passed

github-project-automation Bot moved this from Todo to Done in A2UI Jul 17, 2026


		## Instructions

		1. Audit Codebase Blueprint Compliance:

Uh oh!

Conversation

gspencergoog commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

📋 a2ui-audit Skill

🔍 a2ui-blueprint-compliance Skill

🛠️ Infrastructure & Integration

Impact & Risks

Testing

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

josemontespg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gspencergoog Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gspencergoog commented Jul 16, 2026 •

edited

Loading

📋 `a2ui-audit` Skill

🔍 `a2ui-blueprint-compliance` Skill

gspencergoog Jul 17, 2026 •

edited

Loading