Skip to content

scripts_reference.md

Latest commit

 

History

History
204 lines (182 loc) · 15 KB

scripts_reference.md

File metadata and controls

204 lines (182 loc) · 15 KB
Raw

Script Reference

This page summarizes script entrypoints currently included in the scaffold.

Preservation and integrity

  • scripts/finalize_attempt.py
    • Generates manifest/checksums, writes terminal marker, finalizes attempt directory.
  • scripts/verify_attempt_integrity.py
    • Verifies file checksums for an attempt.
  • scripts/archive_attempt.py
    • Creates a compressed archive for an attempt.
  • scripts/verify_archive.py
    • Validates archive readability and checksum metadata.
  • scripts/audit_preservation.py
    • Audits attempt completeness and marker consistency across data/runs.
  • scripts/validate_run_layout.py
    • Confirms expected top-level run directories exist.

Workflow command stubs

The following are scaffolded placeholders and should be implemented for production execution:

  • scripts/canonicalize_cases.py
  • scripts/register_case.py
  • scripts/run_pf.py
  • scripts/run_dc_opf.py

Campaign input generation

  • scripts/build_pf_anchor_index.py

    • Builds a deterministic Parquet index and checksum manifest from successful AC-OPF samples.jsonl records for downstream PF candidate generation.

  • scripts/generate_pf_candidates.py

    • Generates seeded control-distance, topology, criticality, and response-policy quotas from the anchor index.
    • Streams candidates to an .in_progress JSONL and atomically publishes the completed file, avoiding retention of all expanded candidates in memory.

  • scripts/run_campaign_pf_round.py

    • Validates PF candidate schemas, applies explicit contingency response policies, reuses a persistent PowerModels process, and writes resumable samples and a manifest under <runs_root>/pf/.
    • Validates residuals, controls, voltage/reactive/thermal limits, and exact AC-OPF anchor consistency; preserves each outcome in a separate partition.

  • scripts/reduce_pf_campaign_shards.py

    • Atomically merges PF diversity, boundary, and coverage ledgers across shards and aggregates active-constraint counts while retaining prior rounds.

  • scripts/create_operating_point.py

    • Generates structured operating-point candidates for adaptive campaigns (parametric perturbations plus reference load snapshots when a snapshot registry exists).

  • scripts/enumerate_contingencies.py

    • Enumerates physically credible contingency candidates (N-1 and sequential N-1-1) for a case.

  • scripts/screen_contingencies.py

    • Applies multi-trigger screening and audit tagging to enumerated contingency candidates.

Consistency and gating

  • scripts/compare_solver_consistency.py
    • Compares ExaGO and pandapower AC-OPF outputs on MATPOWER cases and writes a consistency report.
  • scripts/phase1_gate.py
    • Phase-1 gate combining solver-consistency and preservation-audit checks.

Implemented workflow command:

  • scripts/run_ac_opf.py
    • Runs AC-OPF through PowerModelsAdapter for selected MATPOWER cases.
    • Creates full preservation-first attempt directories under data/runs/ac_opf/....
    • Writes normalized outputs, validation placeholders, manifests/checksums, terminal marker, and appends run registry records.
  • scripts/run_scopf.py
    • Runs coupled nonlinear AC SCOPF through PowerModelsSecurityConstrained.run_c1_scopf and Ipopt.
    • Accepts a JSON contingency-set object, a JSON array, or campaign JSONL rows with nested contingency objects; case-tagged JSONL rows are filtered for each selected case.
    • Requires static branch or generator N-1 events and rejects sequential or simultaneous N-k inputs before allocating an attempt.
    • Preserves the resolved case and contingency set, raw and normalized results, solver provenance, logs, timing, checksums, terminal marker, and SCOPF run-registry record.
  • scripts/run_exago_ac_opf.py
    • Runs AC-OPF through ExaGO OPFLOW for selected MATPOWER cases.
    • Parses OPFLOW text output into structured raw_result.solution fields (bus, branch, gen) for downstream HydraGNN-style OPF training conversion.
    • Creates full preservation-first attempt directories and appends run registry records with runtime metadata.
  • scripts/run_pandapower_ac_opf.py
    • Runs AC-OPF through pandapower for selected MATPOWER cases.
    • Creates full preservation-first attempt directories and appends run registry records with runtime metadata.
  • scripts/run_campaign_ac_opf_round.py
    • Runs a campaign round of AC-OPF attempts across cases, topologies, operating points, and contingencies via PowerModelsAdapter.
    • Applies operating-point and contingency transforms (including reference load snapshots) to each parsed case before solving.
    • Encodes the applied contingency into the output path via a deterministic contingency_slug level.
    • --resume skips any candidate whose deterministic output directory already has a finalized attempt; the round report then adds skipped_count and computes failure_fraction over solvable (non-skipped) candidates. See Resumable Campaigns.
  • scripts/run_campaign_exago_ac_opf_round.py
    • ExaGO GPU variant of the campaign map worker: same transforms, ledgers, resume, and reporting as run_campaign_ac_opf_round.py, but solves each candidate with ExaGO opflow.
    • Serializes each transformed case back to a MATPOWER .m netfile (write_matpower_case) so ExaGO solves the identical reduced network PowerModels sees.
    • --solver-mode selects gpu_then_ipopt (default), gpu_only, or ipopt_only; GPU uses HIOPSPARSEGPU/PBPOLRAJAHIOPSPARSE with IPOPT fallback.
    • Captures the power-balance duals (mult_Pmis/mult_Qmis) on every bus from the ExaGO JSON export (LAM_P/LAM_Q) or the stdout summary fallback, matching the single-case and PowerModels paths.
    • Records a normalized feasibility_label (feasible/infeasible/indeterminate/error) on each sample and round-report row; both feasible and infeasible solves are persisted (see Schema Contracts).
    • opflow is launched under a ulimit -c 0 / HSA_ENABLE_COREDUMP=0 guard so a GPU or IPOPT abort on an infeasible case fails cleanly instead of writing a multi-GB core/gpucore.* file; the worker then falls back to IPOPT.
    • Environment: set PGDF_EXAGO_VERBOSE=1 for maximum ExaGO/HiOp verbosity, which also writes one per-shard solver_verbose.log (each attempt's stdout/stderr) next to samples.jsonl; set PGDF_EXAGO_SRUN_PREFIX to launch each opflow as its own singleton Slurm step.
    • Requires a Python env with pyarrow for Parquet ledgers (see Setup); driven by configs/slurm/frontier_exago_acopf_mapreduce_8n_2h.sbatch.
  • scripts/register_load_snapshots.py
    • Discovers per-scenario MATPOWER snapshot files for a case and registers them as reference load operating points.
    • Writes data/operating_point_registry/<case_id>/load_snapshots.json with season, voltage regime, difficulty, and per-bus loads.
  • scripts/shard_selected_candidates.py
    • Deterministically shards selected-candidate JSONL files.
    • Produces fixed shard files and a manifest for map-stage parallel execution.
    • Supports coverage gates (for example by dataset/topology) and deterministic pool-based backfill before sharding.
    • --assignment contiguous --stream creates balanced consecutive input ranges with bounded memory; the Riker PF launcher uses this mode.
  • scripts/reduce_campaign_shards.py
    • Deterministically merges shard campaign ledgers and shard AC execution reports back into the target campaign.
    • Aggregates active-constraint ledger rows by (constraint_family, component_id) and writes a reduce report marker per round.
  • scripts/launch_ultrascale_campaign.py
    • Plans or submits chained map/reduce Slurm rounds with full parameter exports (budget, sharding, coverage gate, backfill, solver runtime settings).
    • Supports dependency-chained submission (afterok) for continuous multi-round campaigns.
  • scripts/drive_campaign.py
    • Auto-detects the furthest incomplete round (lowest round whose reduce marker is missing or not ok) and resubmits it with RESUME=1 so the job resumes interrupted work.
    • --chain queues all subsequent rounds, each chained via afterok, for unattended multi-round completion.
    • Splits budgets across rounds (--total-budget with --budget-schedule) and passes through extra env with repeatable --set KEY=VALUE; --nodes/--ntasks-per-node/--cpus-per-task/--time override the sbatch header; --dry-run prints the plan without submitting. See Resumable Campaigns.
  • scripts/monitor_campaign.py
    • Read-only progress dashboard for a map/reduce campaign round: bootstrap sub-stage, candidate counts, shards done/total, aggregated solved/failed/skipped, and reduce status.
    • Reads only cheap signals (intermediate files, the per-shard reports, and queue/done markers), so it stays fast even with millions of attempt directories. --watch N refreshes every N seconds. See Resumable Campaigns.
  • scripts/analyze_campaign_diversity.py
    • Streams authoritative per-shard diversity ledgers and writes a reproducible JSON, CSV, and self-contained HTML audit under data/reports/diversity/.
    • Computes exact low-cardinality structural coverage and deterministic sampled estimates for robust nearest-neighbor distance, near-duplicate rate, effective sample ratio, intrinsic dimension, active-signature diversity, and similarity-cluster concentration.
    • Uses versioned defaults from configs/diversity_analysis.yaml; see Campaign Diversity Analysis.
  • scripts/analyze_campaign_diversity_mpi.py
    • MPI map/reduce variant of the diversity audit. Deterministically partitions shard ledgers, merges exact counters and the global lowest-hash sample, and writes only from rank 0.
    • Produces the same report schema as the serial reference implementation; use configs/slurm/andes_diversity_analysis_mpi.sbatch for the production corpus.

Source onboarding helpers

  • scripts/inspect_sources.py
    • Reports configured source presence, git head (for git repos), direct-download file presence, and manual-case raw file status.
  • scripts/download_sources.py
    • Clones missing git-based sources from configs/sources.yaml.
    • Supports pinned git checkout and recursive clone configuration.
    • Downloads direct URL artifacts with retries/resume (curl --continue-at -) when --download-files is enabled.
    • Skips already-verified archives and records source URLs, timestamps, SHA-256, and sizes.
    • Optionally extracts archives and writes per-source source_manifest.yaml.
    • Writes provenance report JSON (default: data/imported/source_provenance.json).
  • scripts/prepare_manual_downloads.py
    • Creates TAMU case-folder structure and emits external/tamu/MANUAL_DOWNLOADS.md.
    • Writes external/tamu/manual_download_manifest.yaml with per-case acquisition status.
  • scripts/register_manual_download.py
    • Registers a manually downloaded archive for a specific case.
    • Preserves the original archive under external/tamu/<case_id>/raw/, computes SHA-256, and writes inventory/checksum files.
    • Optionally extracts archive contents and writes case-level source_manifest.yaml.
  • scripts/validate_sources.py
    • Validates configured sources and emits policy statuses.
    • Supports statuses: MISSING, DOWNLOADED_UNREGISTERED, REGISTERED, EXTRACTED, VALIDATED, CHECKSUM_MISMATCH, UNSUPPORTED_FORMAT, INCOMPLETE_FOR_PF, INCOMPLETE_FOR_OPF.
    • Marks validation non-OK when required manual cases are not validated.
  • scripts/audit_case_sources.py
    • Audits per-case source identity metadata and local availability from configs/sources.yaml.
    • Reports source lineage, acquisition mode, expected source file, checksum, TAMU correspondence, and PF/DC-OPF/AC-OPF readiness.
    • Enforces explicit non-equivalence metadata for similar-size but distinct grid families.
  • scripts/export_manual_source_bundle.py
    • Exports a git-tracked reproducibility bundle from manual TAMU case ingestion state.
    • Copies per-case manifests, checksums, and inventories into data/imported/manual_sources/cases/.
    • Writes data/imported/manual_sources/repro_bundle.json with archive checksums and metadata.
  • scripts/convert_go_challenge_to_matpower.py

Topology creation helper

  • scripts/create_topology.py
    • Parses MATPOWER-format case files (currently default-mapped for pglib_opf).
    • Emits a topology JSON artifact under data/topology_registry/<case_id>/topology_<index>_<description>.json.
    • Appends a registry record to data/topology_registry/topology_registry.jsonl with source file and element counts.

Build and machine-profile helpers

  • scripts/install_hsl_ipopt.sh
    • Builds a licensed Coin-HSL archive with Meson against the Riker Julia LP64 OpenBLAS artifact and installs it under private $HOME/.local/coinhsl storage.
    • Generates current/env.sh, explicitly configures Ipopt's hsllib, and validates both MA27 and MA57 before activating the version.
  • configs/slurm/submit_riker_pf_smoke.sh
    • Submits the one-node PF smoke after validating the private Coin-HSL activation. Each candidate tries default Ipopt, MA27, then MA57 as needed.
  • configs/slurm/submit_riker_pf_large.sh
    • Starts the Riker PF campaign supervisor after the HSL preflight. Run only after the smoke completes.
  • configs/slurm/riker_pf_campaign_supervisor.sbatch
    • Submits the first incomplete PF round with resume enabled, then resubmits itself afterany that job. It resumes timeouts, advances after successful reduction, and stops after all configured rounds complete.
  • scripts/configure_exago_build.py
    • Configures machine-scoped ExaGO build/install directories under external/ExaGO/builds/<profile>/.
    • Supports optional configure/build/install execution and emits resolved paths in JSON.
  • scripts/run_exago_andes_opflow.sh
    • Loads the known-good Andes module stack and runs isolated ExaGO opflow.
    • Defaults to case9 IPOPT smoke test when no CLI arguments are provided.
    • Accepts custom opflow arguments for alternate netfiles and solver options.
  • scripts/setup_frontier_venv.sh
    • Builds the canonical campaign virtualenv (.venv) with pandas+pyarrow from requirements.txt, matching the Frontier runtime interpreter.
    • Consumed by the Frontier ExaGO campaign sbatch via PGDF_VENV. See Setup.

CLI module entrypoints

Equivalent module-backed entrypoints are exposed via package scripts:

  • pgdf-finalize-attempt
  • pgdf-verify-attempt
  • pgdf-audit-preservation
  • pgdf-validate-layout