You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/cx/BENCHKIT_GAP_ANALYSIS.md
+34Lines changed: 34 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -57,6 +57,40 @@ BenchKit already has a solid foundation for continuous benchmarking, especially
57
57
Continuous estimation has now moved beyond a mere entry point: a common estimation flow, a `weakscaling` reference path, a detailed dummy package for `qws`, and result-provenance handoff have all been implemented.
58
58
However, estimation is still not yet broadly deployed across multiple applications, and AI-driven optimization integration remains mostly at the integration-point stage.
59
59
60
+
## 2.1 現時点で明示しておく設計負債 / Explicit Design Debts to Keep Visible
At the current stage, several issues are not simply “missing implementations” but rather intentionally deferred or not yet fully fixed as design boundaries.
79
+
This section keeps those visible so they are not forgotten later.
80
+
81
+
- estimation package / flow boundary:
82
+
the separation between BenchKit common flow and package-owned behavior is much clearer now, but metadata discovery, generalization across multiple detailed packages, and package-level comparison are not yet fixed.
83
+
- application-side estimation declaration API:
84
+
the direction of declaring current / future packages and section bindings in `estimate.sh` is becoming clear, but the final cross-application API is not yet frozen.
85
+
- estimation compare UI:
86
+
detail views now expose current / future breakdown, fallback, and applicability, but same-`code`/`exp` comparison remains intentionally deferred.
87
+
- result quality handling:
88
+
quality badges and detail views are implemented, but they are not yet enforced as CI-failing validation; the current approach remains visibility-first.
89
+
- site capability checker:
90
+
`/results/usage` now has lightweight configuration checks, but there is still no automatic checker directly tied to CI execution readiness.
91
+
- app/system coverage evaluation:
92
+
the current coverage matrix uses `list.csv` plus mention-based detection in `build.sh` / `run.sh`, leaving room for a stricter future capability definition.
93
+
60
94
## 3. 機能別ギャップ分析 / Function-by-Function Gap Analysis
Among the open items above, the following are explicitly considered “not fixed yet” at the current stage.
652
+
653
+
- generalization across multiple detailed packages:
654
+
`instrumented_app_sections_dummy` is treated as a reference implementation, while discovery / comparison / fallback rules across multiple detailed packages remain intentionally unfixed.
655
+
- deeper use of package metadata discovery:
656
+
metadata is already owned by packages, but how aggressively it should drive portal behavior or comparison UI is still open.
657
+
- the final compare UI shape:
658
+
current / future detail display exists, but a stable multi-estimate comparison interface is intentionally deferred.
659
+
- fully specified acquisition-path contracts:
660
+
the concepts of `instrumented_app_sections`, `instrumented_tool_sections`, and `counter_based` remain, but their strict capability contracts are not yet frozen.
0 commit comments