Skip to content

Add Monorail telemetry for app config validate#7120

Open
dmerand wants to merge 3 commits intodlm-app-validate-malformed-extension-configsfrom
dlm-app-validate-monorail
Open

Add Monorail telemetry for app config validate#7120
dmerand wants to merge 3 commits intodlm-app-validate-malformed-extension-configsfrom
dlm-app-validate-monorail

Conversation

@dmerand
Copy link
Contributor

@dmerand dmerand commented Mar 26, 2026

What

Add Monorail instrumentation for shopify app config validate.

This records whether --json was used and captures aggregate validate results: whether the run was valid, how many issues were found, and how many files those issues came from.

WHAT is this pull request doing?

  • add cmd_app_validate_json
  • add cmd_app_validate_valid
  • add cmd_app_validate_issue_count
  • add cmd_app_validate_file_count
  • record those fields for both normal validate results and LocalConfigError failures that happen before validateApp runs
  • document the current lightweight agent-identification convention via SHOPIFY_* environment variables without adding new first-class Monorail schema fields yet

Why

We want to measure how app config validate is being used and what kinds of results it produces.

That gives us a way to track JSON usage and validate outcomes while the current validate work rolls out, without introducing a broader telemetry model for agent identity yet.

How

The command records cmd_app_validate_json at entry.

The validate flow records aggregate outcome metadata from canonical validation issues so the counts stay aligned with real validation failures instead of display-only wrapper messages.

Testing

To test manually:

  1. Run shopify app config validate
  2. Run shopify app config validate --json
  3. Run against an app with an invalid config and confirm the command still behaves the same while emitting the new aggregate metadata

Post-release steps

None.

Measuring impact

How do we know this change was effective? Please choose one:

  • n/a - this doesn't need measurement, e.g. a linting rule or a bug-fix
  • Existing analytics will cater for this addition
  • PR includes analytics changes to measure impact

Checklist

  • I've considered possible cross-platform impacts (Mac, Linux, Windows)
  • I've considered possible documentation changes

Copy link
Contributor Author

dmerand commented Mar 26, 2026

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@dmerand dmerand changed the title Introduce AppConfigurationAbortError for app validate json Add Monorail telemetry for app config validate Mar 26, 2026
@dmerand dmerand marked this pull request as ready for review March 26, 2026 21:45
@dmerand dmerand requested a review from a team as a code owner March 26, 2026 21:45
@github-actions
Copy link
Contributor

Differences in type declarations

We detected differences in the type declarations generated by Typescript for this branch compared to the baseline ('main' branch). Please, review them to ensure they are backward-compatible. Here are some important things to keep in mind:

  • Some seemingly private modules might be re-exported through public modules.
  • If the branch is behind main you might see odd diffs, rebase main into this branch.

New type declarations

We found no new type declarations in this PR

Existing type declarations

packages/cli-kit/dist/public/node/monorail.d.ts
@@ -66,6 +66,10 @@ export interface Schemas {
             cmd_app_linked_config_uses_cli_managed_urls?: Optional<boolean>;
             cmd_app_warning_api_key_deprecation_displayed?: Optional<boolean>;
             cmd_app_deployment_mode?: Optional<string>;
+            cmd_app_validate_json?: Optional<boolean>;
+            cmd_app_validate_valid?: Optional<boolean>;
+            cmd_app_validate_issue_count?: Optional<number>;
+            cmd_app_validate_file_count?: Optional<number>;
             cmd_dev_tunnel_type?: Optional<string>;
             cmd_dev_tunnel_custom_hash?: Optional<string>;
             cmd_dev_urls_updated?: Optional<boolean>;

@github-actions
Copy link
Contributor

We detected some changes at packages/*/src and there are no updates in the .changeset.
If the changes are user-facing, run pnpm changeset add to track your changes and include them in the next release CHANGELOG.

Caution

DO NOT create changesets for features which you do not wish to be included in the public changelog of the next CLI release.

@github-actions
Copy link
Contributor

Coverage report

St.
Category Percentage Covered / Total
🟢 Statements 82.39% 15237/18494
🟡 Branches 74.9% 7513/10031
🟢 Functions 81.49% 3839/4711
🟢 Lines 82.77% 14395/17391

Test suite run success

4016 tests passing in 1530 suites.

Report generated by 🧪jest coverage report action from 92db345

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant