Projections: .sts provenance is incomplete (PROJECTIONS_ID empty; COMMANDLINE misses pre-snapshot args; no app version / job id / environment)

[lvkale]

Summary

The Projections .sts file records useful run provenance (USERNAME, HOSTNAME, COMMANDLINE, TIMESTAMP, CHARMVERSION, MACHINE, PROCESSORS, SMPMODE), written by traceWriteSTS() in src/ck-perf/trace-common.C. In practice, though, a .sts is not sufficient to identify the run that produced a trace. Five gaps, one of which is a straightforward bug.

This came up while comparing several traces of the same application at 480 PEs, where telling two runs apart after the fact turned out to be impossible from the trace files alone.

1. PROJECTIONS_ID is always empty (bug)

src/ck-perf/trace-projections.C, in LogPool::writeSts():

  // generate an automatic unique ID for each log
  fprintf(stsfp, "PROJECTIONS_ID %s\n", "");

The comment states the intent; the code emits an empty string. The one field designed to identify a run carries no information. A UUID, or a hash of (hostname, pid, start time), would make traces self-identifying and would let tools detect that two directories hold the same run.

2. COMMANDLINE is a post-stripping snapshot, with no indication

Cmi_argvcopy is captured after the SMP arguments have already been consumed and deleted from argv:

• mainline: +ppn parsed at src/arch/util/machine-common-core.C:1227, +p at 1231, snapshot at 1412
• reconverse: +pe/+p/+ppn parsed at convcore.cpp:278-283, snapshot at 296

Since CmiCopyArgs() copies the pointer array, arguments deleted before the snapshot are absent and arguments deleted after it survive. Observed on a 480-PE SMP run launched as

srun ... ./FoF3 -f <data> -d oct -u dist -b 0.2 +ppn 15 +traceroot <dir>

the recorded line is

COMMANDLINE "/…/FoF3 -f <data> -d oct -u dist -b 0.2 +traceroot <dir>"

+ppn 15 is gone; +traceroot (parsed later) is kept. SMPMODE 15 32 happens to recover the launch shape in this case, so the practical damage is limited — but the field silently misrepresents how the program was invoked, and anything else consumed that early would vanish the same way. Taking the snapshot before any parsing would fix it.

3. No application version

CHARMVERSION records charm's own git describe. Nothing records the application's revision, so a trace cannot say which version of the user's code produced it — usually the first question when comparing two traces of the same program. An optional API for the application to register a version string (recorded as e.g. APPVERSION) would cover this.

4. No allocation / job identity

HOSTNAME is the hostname of the writing PE's node only — one node of however many. There is no node list and no scheduler job identifier. On clusters where allocation-to-allocation performance variation is significant, knowing whether two traces came from the same allocation matters before their timings are compared at all. Recording a node list, and the scheduler job id when present in the environment (SLURM_JOB_ID, PBS_JOBID, LSB_JOBID, …), would address it.

5. No environment capture

Nothing records the dynamic library environment. A binary built against a tracing-enabled Charm but launched with LD_LIBRARY_PATH pointing at a non-tracing runtime produces a normal-looking .sts, and nothing in the trace reveals the mismatch. Recording LD_LIBRARY_PATH (or the resolved paths of the loaded Charm/Converse libraries) would make that diagnosable after the fact.

Suggested priority

(1) and (2) look like small, self-contained fixes — fill in the ID, and move the argv snapshot ahead of parsing. (3)-(5) are additive fields.

Happy to put up a PR for (1) and (2) if the approach sounds right.

Environment: charm v8.0.1-devel-113-g9460a3469, Projections .sts VERSION 11.0, Linux x86_64, SMP.