splunkenizer · Oct 28, 2026 · Oct 28, 2026 · Oct 28, 2026 · Oct 28, 2026
diff --git a/.agent/workflows/deploy-aws-terraform.md b/.agent/workflows/deploy-aws-terraform.md
@@ -1,120 +1,25 @@
 ---
-description: Deploy Splunk infrastructure to AWS using Terraform
+description: Legacy pointer. AWS Terraform deploy now runs through the /spa and /spa-create-config skills.
 ---
 
-# Deploy Splunk Infrastructure to AWS with Terraform
+# Deploy Splunk to AWS (legacy workflow)
 
-This workflow provisions AWS infrastructure using Terraform and deploys Splunk. Use **only `spa`** from an environment directory (`SPA_ENV_DIR`). Do not call `ansible-playbook` or `python3 bin/…` here.
+This file is a legacy runbook. The skills own the flow now. Agents use `/spa-create-config` to design and write `config/splunk_config.yml`, and `/spa` for every lifecycle command. Do not follow step lists from older copies of this file.
 
-## Prerequisites
+## Rules that apply
 
-1. **Environment selection:** use `spa --env NAME`, set a registered default, or `cd` into the env. The installed `spa` launcher is on `PATH`.
-2. **Host tools:** `spa doctor` (Terraform for AWS). Ansible collections come from `spa_venv` — do not run `ansible-galaxy` in this workflow.
-3. **AWS credentials:** `spa aws --check-auth --json`. Report set/not-set only; never print secret values.
-4. **Config:** `config/splunk_config.yml` with `terraform.aws` (security group, key pair, AMI in the target region).
+- Use only `spa` from the environment directory (`SPA_ENV_DIR`), or pass `--env NAME`. Do not call `ansible-playbook`, `terraform`, or `python3 bin/…` directly.
+- Never edit `config/splunk_config.yml` by hand. Change it with `spa config …`.
+- `provision`, `deploy`, `destroy`, `suspend`, `resume`, and `environment remove` each wait for an explicit yes in chat for that run. A request in the same message is not that yes. Then pass `--yes`.
+- Do not pass `-v` or `--ansible-output` to `provision`, `deploy`, or `run`.
+- Use full command names: `spa hosts ssh <host>`, not `spa sh`.
 
-## Step 1: Configure Infrastructure
+## Where each step lives
 
-Edit `config/splunk_config.yml` and add/update the `terraform.aws` section (see examples). Discover AMIs and resources with `spa aws --survey --region <region> --json` when credentials work.
-
-## Step 2: Preview Infrastructure Changes
-
-```bash
-spa provision --yes -- --tags plan
-```
-
-This generates `terraform/aws/terraform.tfvars`, runs `terraform init` / `plan`, and does **not** apply.
-
-## Step 3: Provision Infrastructure
-
-```bash
-spa provision --yes
-```
-
-Non-interactive (agents / CI): `spa provision --yes && spa deploy --yes`. `--yes` after the subcommand confirms the action (`-y` is the short form; `spa -y provision` also works). Agents must pass `--yes`; they never answer the human `Proceed? [y/N]` prompt.
-
-This will:
-
-1. Generate Terraform configuration
-2. Initialize Terraform
-3. Show the plan
-4. Prompt for confirmation unless `--yes`
-5. Create EC2 instances and `inventory/hosts`
-6. Wait for SSH by default
-
-Skip SSH wait: `spa provision --yes -- -e wait_for_ssh=false`.
-
-## Step 4: Verify Host Readiness (optional)
-
-```bash
-spa run aws_wait_hosts
-```
-
-SSH, Python, uptime, cloud-init. Success text in the playbook may mention deploy; continue with `spa deploy`.
-
-## Step 5: Deploy Splunk
-
-```bash
-spa deploy
-```
-
-## Step 6: Verify Deployment
-
-Open `$SPA_ENV_DIR/config/index.html`, or:
-
-```bash
-spa run create_linkpage
-```
-
-SSH: `spa hosts ssh <host>` or `spa sh <host>` when the host is in inventory.
-
-## Pause and Resume
-
-Stop EC2 compute while retaining Terraform state and EBS volumes:
-
-```bash
-spa suspend --yes
-```
-
-Resume waits for AWS instance health and rewrites `inventory/hosts` with the
-new public DNS names/IPs before returning:
-
-```bash
-spa resume --yes
-```
-
-This is not a destroy operation, but EBS volumes and other retained resources
-continue to incur charges. In agent mode both commands require `--yes`.
-
-## Troubleshooting
-
-- `spa doctor` and `spa aws --check-auth --json` first.
-- Verbose wait: `spa run aws_wait_hosts -- -v`
-- Longer SSH timeout: `spa run aws_wait_hosts -- -e ssh_timeout=600`
-
-## Cleanup
-
-```bash
-spa destroy
-```
-
-In agent mode, destroy requires `-y` / `--yes`. This permanently deletes EC2 instances.
-
-## Quick Reference
-
-| Command | Purpose |
-|---------|---------|
-| `spa doctor` | Host prerequisites |
-| `spa aws --check-auth --json` | Credential probe (no secret values) |
-| `spa provision --yes -- --tags plan` | Preview changes |
-| `spa provision --yes && spa deploy --yes` | Provision then deploy (stop if provision fails) |
-| `spa provision --yes` | Provision infrastructure only |
-| `spa run aws_wait_hosts` | Verify host readiness |
-| `spa deploy` | Deploy Splunk (hosts already provisioned); `--hosts` for a subset |
-| `spa suspend --yes` | Stop EC2 instances; keep state and disks; optional `--hosts` |
-| `spa resume --yes` | Start EC2 instances; wait and refresh inventory; optional `--hosts` |
-| `spa run create_linkpage` | Host link page |
-| `spa destroy` | Destroy infrastructure |
-| `spa hosts list --status` | Inventory names, roles, runtime state |
-| `spa hosts ssh <host>` / `spa sh <host>` | SSH via inventory |
-| `spa run list` | Playbook catalog |
+| Step | Owner |
+|------|-------|
+| Design, AWS settings, and config writes | `/spa-create-config` |
+| Credential probe | `spa aws --check-auth --json` (report set / not set only) |
+| Preview, provision, host check, deploy, suspend, resume, destroy | `/spa` |
+| Host status and SSH | `spa hosts list --status`, `spa hosts ssh <host>` |
+| Logs | `spa logs --last` |
diff --git a/AGENTS.md b/AGENTS.md
@@ -6,7 +6,8 @@ SPA provisions and deploys Splunk Enterprise (AWS Linux + Terraform, or VirtualB
 
 - **`SPA_HOME`**: framework (prefix or clone). **`SPA_ENV_DIR`**: one environment. Operator `spa` needs `spa init` first. Path diagram: [user guide](docs/user-guide.md#understand-the-paths).
 - Call **`bin/spa`** only (`spa agent schema`, `spa --agent config search QUERY`, `spa --agent apps`). Load **`/spa`** first ([skills/spa/spa/SKILL.md](skills/spa/spa/SKILL.md)). Do not import `spa.api`. Do not start a daemon. Flag catalog: [docs/commands.md](docs/commands.md).
-- Mutating commands need **`--yes`**. Never auto-run provision, deploy, suspend, resume, or destroy. Never answer `Proceed?` interactively.
+- Do not hand-edit `splunk_config.yml`. Change it with `spa config ...` (skill `/spa-create-config`) and add apps with `spa apps ...` (skill `/spa-apps`).
+- Mutating commands need **`--yes`**. Never auto-run provision, deploy, suspend, resume, or destroy. Exception: `/spa-create-config` Phase 9 option A hands off to `/spa`, which runs `provision`, then `deploy`, each only after its own explicit yes in chat. Never answer `Proceed?` interactively.
 - Never print secrets (Splunkbase, AWS keys, vault). Report env vars as **set / not set**. YAML: `lookup('env', ...)`. Do not pass `-v` on provision/deploy/run (unredacted Ansible stdout).
 - Full command names (`spa hosts ssh`, `spa validate`), not `spa sh` / `spa val`.
 - Playbooks: `spa --agent run list` then `spa run NAME --help`. Confirmation: `--yes`. Extra Ansible args: `spa run NAME -- …`.
@@ -17,14 +18,14 @@ Human loop (same as README): [docs/user-guide.md](docs/user-guide.md).
 
 | Task | Command / path |
 | --- | --- |
-| Config | `$SPA_ENV_DIR/config/splunk_config.yml` via `spa init --example` |
+| Config | `$SPA_ENV_DIR/config/splunk_config.yml` via `spa init --example TOPOLOGY --provider aws|virtualbox|external NAME` |
 | Schema vs catalog | Pydantic (`spa validate`); `spa config search` then `spa config explain PATH` |
 | New config key | [docs/contributing.md](docs/contributing.md#adding-a-configuration-feature) |
 | All settings | [docs/config.md](docs/config.md) (`spa --no-agent config schema --markdown`) |
 | CLI flags | [docs/commands.md](docs/commands.md) (`spa --no-agent agent schema --markdown`) |
 | Guided design | [docs/user-guide.md](docs/user-guide.md#configure-an-environment) · `spa config search` then `spa config explain` |
 | 2.x → 3.0 | [docs/migrate.md](docs/migrate.md) |
-| New env | `spa environment init --example cm_2idxc_sh NAME` then `spa --env NAME …`, `spa environment set --default NAME`, or `cd` into it |
+| New env | `spa environment init --example cm_2idxc_sh --provider aws NAME` then `spa --env NAME …`, `spa environment set --default NAME`, or `cd` into it |
 | Features / apps | `spa --agent config search QUERY`; `spa --agent apps search QUERY` then `snippet APP_ID`. For 5391: `spa --agent apps content-packs list 5391`. AWS live values: `spa aws --json` |
 | Python | `spa venv create --shared --yes` (`spa init` creates it when missing) |
 | Repair Python venv | `spa venv reinstall --shared --yes` (or `spa venv rebuild --shared --yes`) |
@@ -52,12 +53,11 @@ Canonical: `skills/spa/`. When to load:
 
 | Skill | When |
 | --- | --- |
-| [spa](skills/spa/spa/SKILL.md) (`/spa`) | Load first: Software, env lifecycle, operator loop |
+| [spa](skills/spa/spa/SKILL.md) (`/spa`) | Load first: Splunk lab or environment request (any topology, OS, or CPU count), Splunk deploy to existing hosts, Software, env lifecycle, operator loop. Use spa, not the AWS CLI or Terraform. |
 | [spa-create-config](skills/spa/spa-create-config/SKILL.md) | `splunk_config.yml` in an existing env |
 | [spa-apps](skills/spa/spa-apps/SKILL.md) | Splunkbase search / snippet / download |
-| [spa-add-test-scenario](skills/spa/spa-add-test-scenario/SKILL.md) | App-scope tests |
 
-Cursor: `/spa`, `/spa-create-config`, `/spa-apps`, `/spa-add-test-scenario` after `spa skills install`. See [docs/contributing.md](docs/contributing.md#agent-skills).
+Cursor: `/spa`, `/spa-create-config`, `/spa-apps` after `spa skills install`. See [docs/contributing.md](docs/contributing.md#agent-skills). App-scope scenarios are in [tests/README.md](tests/README.md).
 
 Backend vs GUI: [docs/architecture.md](docs/architecture.md#controller-architecture).
 

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -9,8 +9,25 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 Work on this branch is **3.0** (M1–M3 plus the follow-on issues below). Linux packages and Homebrew are later, not part of 3.0.
 
+### Changed
+
+- **Validate flag renamed**: `spa validate --splunk-config-aws` is now `spa validate --splunk-config aws`. The old flag is removed, so scripts that pass it must switch to the new form.
+
+- **Skill gate for new environments**: `/spa-create-config` no longer writes config until the location, name, and summary gate passes. A new-environment request never selects an existing environment, and a write to a registered environment needs that environment named first. Each config or apps write checks `resolved_env_name` in its response. Provision, deploy, host check, destroy, suspend, resume, and logs now run from `/spa` after Phase 9 option A hands off there.
+
+- **ITSI search groups results**: `spa apps search itsi` in human mode groups hits into "Core ITSI" (1841), "Content packs (from Library 5391)", "Single content packs", and "Other apps". The library section shows the `content-packs list 5391` and `--content-pack NAME` commands, and each single pack shows its add command and that Core ITSI 1841 is added first when missing. Library 5391 is pinned for ITSI queries, so its row appears even when the page omits it. Agent output and non-ITSI searches are unchanged.
+
+- **terraform.aws rejects unknown keys**: a misspelled key such as `ami` next to `ami_id` now fails validation instead of being ignored.
+- **AWS discovery uses the environment region**: `spa aws` without `--region` searches `terraform.aws.region` from the environment config before `AWS_REGION`, `AWS_DEFAULT_REGION`, or `~/.aws/config`. The skills pass `--region` on every AMI and survey lookup.
+
+- **Environment cleanup**: `spa environment cleanup` lists the registered environments whose directory is missing and removes only their registry entries after a yes (`--yes`, or a prompt on a terminal). It deletes no directories. With `--yes` it prints each entry it removed. A removed default moves to the first remaining environment.
+
+- **Init needs a provider**: `spa init` and `spa environment init` require `--provider aws|virtualbox|external` unless a default is saved with `spa environment set --provider NAME`. With neither, init stops before it writes anything and names the three choices. Saving `aws` as the default now writes `providers.yml` (it used to delete the file), so an explicit `aws` counts as a choice. In agent mode, a failed init returns `ok` false and `error` only, with no `data`.
+
 ### Added
 
+- **Skills pack** ([#67](https://github.com/splunk/splunk-platform-automator/issues/67)): three operator skills (`spa`, `spa-create-config`, `spa-apps`). Agents ask in chat and wait before provision, deploy, destroy, suspend, resume, and environment remove. `spa validate --splunk-config aws` checks that provider from the environment config. App-scope scenario steps live in `tests/README.md`.
+
 - **External register addresses**: `spa hosts register` takes `--ssh-host` and `--ssh-port` for the SSH connection, and optional `--ip-addr` for the address other Splunk hosts use. The SSH host, port, user, key, and any `--ip-addr` are written only in `inventory/hosts`. Omit `--ip-addr` to use the SSH host for both. Omit `--ssh-port` to use port 22. The host link page keeps the public DNS name or the SSH host in the URL.
 
 - **Agent results** ([#67](https://github.com/splunk/splunk-platform-automator/issues/67)): `spa licenses` and `spa aws --validate` return a short JSON result when an agent is detected, and the full objects with `--details`. `spa aws --validate` warns when a named security group allows `0.0.0.0/0` or `::/0` to ports 22, 80, 8000, 8088, or 8089. An agent run of `provision`, `deploy`, `destroy`, and `run` prints one plain sentence on stderr when the phase or host changes, and the heartbeat writes nothing. A host is named once. Extracting the Splunk archive is its own sentence. Host preparation says it is already done when that phase changes nothing. A role says it is already configured only after that role's own tasks finish with nothing changed; the opening checks do not count. A role play with no matching hosts is silent. Provision, deploy, and destroy share one lock. While one is running, the others are refused. The lock ends when that process ends, including when it is aborted. `spa logs --follow` stops on Ctrl+C without a traceback. `Planning the infrastructure.` covers the plan, and `HOST is up.` follows the instance readiness result. The Terraform plan summary is not a progress JSON line. `splunk_download` says the host downloads the tarball when the flag is true and the controller uploads it from Software when the flag is false.

diff --git a/ROADMAP.md b/ROADMAP.md
@@ -57,7 +57,8 @@ Modeled on [cup](https://github.com/splunk/cup) agent mode (detect agent env, JS
 - [ ] Hide credentials in Ansible output (`no_log`, stop putting secrets on `command:` argv) so `-v` can stay native debug without leaking admin/Splunkbase/SHC/API secrets; keep agents off the transcript — [#88](https://github.com/splunk/splunk-platform-automator/issues/88)
 - [x] Main `spa` skill (cup-style `/spa` entry) plus `spa environment` registry — [#54](https://github.com/splunk/splunk-platform-automator/issues/54)
 - [x] `spa skills install` / `spa skills remove` (Claude / Cursor / Codex and the other cup IDE paths, plus Antigravity) from `$SPA_HOME/skills/spa/` — [#55](https://github.com/splunk/splunk-platform-automator/issues/55)
-- [ ] Skills, agents, and runbooks pack (full operator loop; calls `spa` only). Add-app skills use Splunkbase/vendor **install docs** (when linked) to set `target_roles` / `splunk_config.yml`, e.g. [Unix TA Install](https://splunk.github.io/splunk-add-on-for-unix-and-linux/Install/) — [#67](https://github.com/splunk/splunk-platform-automator/issues/67). Blocked by [#55](https://github.com/splunk/splunk-platform-automator/issues/55)
+- [x] Skills, agents, and runbooks pack (full operator loop; calls `spa` only). Add-app skills use Splunkbase/vendor **install docs** (when linked) to set `target_roles` / `splunk_config.yml`, e.g. [Unix TA Install](https://splunk.github.io/splunk-add-on-for-unix-and-linux/Install/) — [#67](https://github.com/splunk/splunk-platform-automator/issues/67). Blocked by [#55](https://github.com/splunk/splunk-platform-automator/issues/55)
+- [ ] Local archive check, AWS lab settings, and apps/upgrade/health references: `splunk_archive_glob` with the missing-file download URLs and `SplunkDownloadConfig` descriptions; the nine AWS lab `spa config set` keys and `spa validate --splunk-config aws` in create-config; Unix TA install docs, `spa config merge itsi-minimal`, Java on the search host, `upgrade.md`, and `health.md` — [#125](https://github.com/splunk/splunk-platform-automator/issues/125). Split from [#67](https://github.com/splunk/splunk-platform-automator/issues/67)
 - [ ] Apps agent interface: search shows where app 833 belongs, a Splunkbase timeout is retryable, `spa apps auth check` reports whether credentials are set, and `spa config merge itsi-minimal` applies the four-app ITSI set — [#124](https://github.com/splunk/splunk-platform-automator/issues/124). Slice of [#67](https://github.com/splunk/splunk-platform-automator/issues/67)
 - [x] Splunkbase search / snippet / download (`spa apps`; app-management skill; never display Splunkbase passwords — [secrets-handling.md](skills/spa/spa-create-config/references/secrets-handling.md)) — [#59](https://github.com/splunk/splunk-platform-automator/issues/59). Config add/remove is [#76](https://github.com/splunk/splunk-platform-automator/issues/76).
 - [ ] Optional [cup](https://github.com/splunk/cup) integration (plugin; spa works without cup; cup uses `spa hosts copy` / `ssh` for file transfer) — [#62](https://github.com/splunk/splunk-platform-automator/issues/62)

diff --git a/ansible/plugins/inventory/schema.py b/ansible/plugins/inventory/schema.py
@@ -96,7 +96,7 @@ class VirtualBoxConfig(BaseModel):
 
 class AwsTerraformConfig(BaseModel):
     """AWS configuration for Terraform provisioning."""
-    model_config = ConfigDict(extra='allow')
+    model_config = ConfigDict(extra='forbid')
 
     region: Optional[str] = Field(None, description="AWS region Terraform uses. Default eu-central-1. Omit the key to keep that.")
     ami_id: Optional[str] = Field(

diff --git a/docs/aws.md b/docs/aws.md
@@ -13,7 +13,7 @@ spa aws --survey --region REGION --json
 spa config explain provider.aws
 ```
 
-`spa aws` uses `--region` when you pass it. Otherwise it uses `AWS_REGION`, then `AWS_DEFAULT_REGION`, then `region` in `~/.aws/config` for the active profile. Text output names that source on stderr, and the listing stays on stdout. `spa aws regions list` marks that region. `--json` and `--agent` print only JSON; the same source is `region_source` in that body, and error text is redacted. `spa aws list regions` exits 2 and names `spa aws regions list`. Provision, suspend, resume, and destroy still use `terraform.aws.region` from the environment config.
+`spa aws` uses `--region` when you pass it. Otherwise it uses `AWS_REGION`, then `AWS_DEFAULT_REGION`, then `region` in `~/.aws/config` for the active profile. Text output names that source on stderr, and the listing stays on stdout. `spa aws regions list` marks that region. `--json` and `--agent` print only JSON; the same source is `region_source` in that body, and error text is redacted. The body also carries `aws_config_region` and `aws_config_profile`, the `~/.aws/config` region even when an environment config sets `terraform.aws.region` first. `spa aws list regions` exits 2 and names `spa aws regions list`. Provision, suspend, resume, and destroy still use `terraform.aws.region` from the environment config.
 
 Configure `terraform.aws` with a region, AMI, EC2 key pair, SSH user/key, security groups, and defaults such as instance type and root volume. Per-host `terraform.aws` overrides global defaults. `iter` can generate numbered hosts. Terraform modules remain in `SPA_HOME`; generated variables and state remain in `SPA_ENV_DIR`.