[Discussion] Rethinking the API test experience

[seminalworks]

We have six to eight different testing interfaces across the product, three of them
third-party, and which one you get depends on your protocol and gateway vendor. The
OpenAPI path uses an embedded Swagger UI that we cannot modify, which means it cannot
use the shared request-builder components the curl console already uses — and in at
least one flow it sends new users to a guaranteed 404 with no way to distinguish it from a
broken service.

I would like to scope a more consistent test experience, most likely targeting the new
console rather than the current one. Before I invest in research I want to know what is
already in flight, what constraints I am missing, and whether this is the right bet
right now.

The problem

It is fragmented. In choreo-console alone: OpenApiConsole (Swagger UI),
CURLConsole (our own curl builder), GraphQLConsole (GraphiQL),
WebSocketTestConsole (ours), MCPInspector (a vendored third-party app carrying its
own design system into a MUI codebase), TryWithAIViewer, plus Quick Start's own two
variants selected on gateway vendor. A user testing two different APIs can see two
unrelated interfaces.

The consolidation has already started, and Swagger UI cannot join it.
modules/testComponent/components/ holds a shared kit — RequestBuilder,
MethodSelector, RelativePath, BaseSelector, SecurityHeaderToken. CURLViewer
uses it. The OpenAPI path cannot, because it is a third-party widget we configure
rather than compose.

It produces a concrete, reproducible failure. A proxy created from an endpoint gets
a /* catch-all, which forwards any path but cannot enumerate one. Swagger UI
generates its Request URL from the operation, so the obvious thing to try returns 404 —
and that 404 is indistinguishable from a broken backend, a bad deploy, or a platform
fault. This lands on the lowest-intent path: default sample, default tab, cloud
gateway.

Early benchmarking suggests we are unusual. Competitors tend to default to a single
ready-to-send request rather than opening on resource selection and parameter
customisation. I want to do this properly rather than assert it, which is part of what
I am asking about below.

Scope I have in mind

Explicitly not removing swagger-ui-react everywhere. It appears in ~13 files
doing three different jobs: rendering a spec during create, viewing/editing a spec, and
actually sending requests. Using a spec renderer to render a spec is fine. Only the
third job is in question — roughly two places.

Questions

1. Does the new console already change this? My understanding is that broader
   onboarding work is deferred to the new UI targeting ~November. If its test
   experience is already designed or built, that changes this from a proposal into
   feedback — and I would rather find out now.
2. Is anything in flight that overlaps? I am aware of Support OpenAPI Specification in API Platform #3396 and the out-of-sync
   warning UI it records; I suspect there is more I have not found.
3. What am I wrong about? I come from consumer SaaS, so I would rather be told
   early which of these are deliberate API-tooling conventions that only look like
   problems to me.
4. Constraints I should know before investing. Protocol coverage, spec-driven
   behaviour we depend on, auth flows, anything that makes a custom console more
   expensive than it looks.
5. Who should be in the room? Looking for a DRI for the new console's test
   experience, and anyone holding customer feedback on testing.

What I would bring next

If this is worth pursuing: an audit of every test surface and its divergences, a
competitive benchmark, and annotated walkthroughs of the current failure modes. I have
screenshots of the divergences and the Quick Start failure already and can share them
on request — I have kept this post to the argument rather than the evidence.

[nuwand]

I think we should build our own API tester UI, without depending on the swagger-ui for it anymore.