Skip to content

refactor: move step templates to source-first authoring under src#1665

Draft
bcullman wants to merge 21 commits intoOctopusDeploy:masterfrom
bcullman:src-step-template-unpack
Draft

refactor: move step templates to source-first authoring under src#1665
bcullman wants to merge 21 commits intoOctopusDeploy:masterfrom
bcullman:src-step-template-unpack

Conversation

@bcullman
Copy link
Copy Markdown
Contributor

@bcullman bcullman commented Mar 28, 2026

Background

This PR addresses the long-standing reviewability problem described in #575.

Previously, step templates lived under /step-templates as packed JSON files with embedded script content. That packed format is still required for compatibility with Octopus and the Library site, but it made review difficult because script changes appeared as escaped JSON string blobs instead of normal source files. Over time that also made Git history and blame much less useful for understanding how templates had changed.

To compensate for that, the repo accumulated helper workflows and scripts for packing, unpacking, and diffing. Those helped, but the authoring experience still centered around generated artifacts instead of source files.

This PR moves the repository to a source-first model:

  • step templates are now authored under /src/step-templates/<template>/
  • each template is represented by a metadata.json file plus sibling source script files such as scriptbody.ps1, scriptbody.sh, scriptbody.py, predeploy.ps1, deploy.ps1, and postdeploy.ps1
  • the packed JSON files under /step-templates/*.json remain generated compatibility output only

The migration used git mv for template metadata files so history remains attached to the authored template sources as much as possible.

Results

This PR delivers four main outcomes:

  1. Step templates are now reviewed as source files.
    Script changes are visible as normal file diffs under /src/step-templates/..., which makes PR review, blame, and history inspection much clearer.

  2. /src/step-templates is now the source of truth.
    /step-templates/*.json is generated-only compatibility output and is ignored by Git.

  3. The website and Library data flow remain compatible.
    The site still builds from generated packed JSON, and the aggregate template data is still produced for the app.

  4. The developer workflow is improved.
    npm run dev and watch mode now support source-first authoring directly, with better generation performance and better live-reload behavior.

Before

  • Templates were authored as packed JSON under /step-templates
  • Script bodies were embedded inside JSON
  • Review of script changes was difficult
  • Template history was difficult to inspect meaningfully
  • Diff helper tooling was needed to make reviews practical
  • Dev startup repacked migrated templates inefficiently
  • Editing source files could rebuild generated JSON, but the running site did not always refresh correctly
  • npm run dev opened the BrowserSync proxy tab on 3000, even if the useful app URL was 9000
  • Template History links on the site could point at broken or legacy-generated paths

After

  • Templates are authored under /src/step-templates/<template>/
  • Script files are stored as real source files beside metadata.json
  • Packed JSON under /step-templates/*.json is generated-only
  • Contributor docs and review guidance now describe the source-first workflow
  • The site still consumes generated packed JSON and continues to build successfully
  • npm run dev startup generation is significantly faster
  • Source edits regenerate the packed JSON and refresh the running app correctly
  • Dev startup opens the app on 9000 while keeping BrowserSync available in the background

Notable Implementation Details

  • Moved shared authored assets from step-templates/logos and step-templates/tests into src/step-templates
  • Preserved the existing _pack.ps1 and _unpack.ps1 behavior as the canonical packing and unpacking implementation
  • Added helper tooling around the unchanged packer for source-first generation
  • Migrated all existing step templates to the source-first layout in batches, with round-trip validation against the original packed files
  • During migration, each original packed JSON file was copied to a temporary .json.orig file before conversion
  • Those .json.orig files were used as the validation baseline after repacking from the new src/step-templates/... source layout
  • The regenerated packed JSON was compared against the saved .json.orig version to prove that the new source-first representation could round-trip back to the original packed artifact
  • The only accepted normalization differences were the historical ExportedAt formatting behavior of the unchanged packer and a trailing final newline difference where present
  • This validation step was critical because it proved the migration changed the authoring model without unintentionally changing the Octopus-compatible packed template payloads
  • Updated .gitignore so generated step-templates/* output is not committed
  • Updated README, contributing guidance, and the PR template to reflect the final workflow
  • Removed migration-only artifacts and temporary migration scripts after cutover
  • Kept .github/workflows/Hyponome.yml in place for now, but it may also no longer serve a meaningful purpose after this change

Additional Fixes Included In This PR

Along the way, this PR also fixes several related issues that surfaced during the migration:

  • Fixed dev-time template data loading so regenerated template data is served without requiring a server restart
  • Fixed step-template startup generation performance by batching full generation through a single pack invocation instead of spawning PowerShell once per template
  • Removed duplicate full-generation work from the dev startup pipeline
  • Fixed dev live-reload so changes to source step templates can refresh the running app automatically
  • Adjusted dev browser-launch behavior so npm run dev opens the primary app URL on 9000 instead of the less-useful proxy tab on 3000
  • Improved dev startup UX so npm run dev waits for the app server to be ready before opening the browser, avoiding the brief connection-refused page on startup
  • Fixed HistoryUrl generation so template history links resolve to the source-authored template location instead of broken generated paths
  • Fixed the Library template History link UX so it opens in a new tab instead of navigating away from the current page

Follow-Up Candidates

The following items were intentionally retained in this branch, but may no longer be necessary after this change:

  • .github/workflows/Hyponome.yml
  • tools/_diff.ps1
  • tools/Converter.ps1 (possible follow-up cleanup if we confirm it is fully redundant with the current pack/unpack and source-first tooling)

Assumptions

  • This repository does not make it clear how content from this repo is published to https://library.octopus.com/. The assumption for this PR is that the existing publication path either already runs the normal build and generation flow, or only needs a small workflow adjustment so generated step-templates/*.json are produced before publication.
  • Octopus community step templates are synchronized from https://library.octopus.com/, not directly from this Git repository. Based on the Octopus documentation, this PR should therefore remain compatible with Octopus as long as the Library site continues to publish valid packed step template JSON.

Pre-requisites

  • Id values were preserved for existing templates
  • Existing step-template parameter names remain namespaced in authored metadata
  • LastModifiedBy fields remain present
  • The best practices documented here have been applied
  • Category logos now live under src/step-templates/logos
  • The humanize category mapping remains in place in gulpfile.babel.js
  • Version should be incremented

Note: template Version values were not broadly incremented in this PR because the intent was repository authoring-model migration and compatibility preservation rather than semantic template behavior changes. If reviewers want version increments purely for republished template sync behavior, that should be called out explicitly because it would intentionally change the update semantics for every migrated template.

Addresses #575

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant