Repository navigation
[OPIK-8831] [FE] [BE] feat: send one media part per list entry from a single Playground variable - #8981
[OPIK-8831] [FE] [BE] feat: send one media part per list entry from a single Playground variable#8981aadereiko wants to merge 4 commits into
Conversation
… single Playground variable
An image, video or audio slot always became exactly one part, so a list
variable could not carry a varying number of images. In the browser a list
rendered as "url1,url2" (or as CSV JSON text) and the provider rejected the
URL; on test-suite runs media URLs were never rendered at all, so the
provider got the literal "{{images}}" or "{{image_1}}".
A slot whose URL is exactly one variable holding a list (a JSON array, or
JSON-array text as CSV uploads store it) now expands into one part per item,
in order; an empty list sends none. Everything else renders as before. The
same rule runs in usePromptDatasetItemCombination (dataset runs) and in
ExperimentMessageRenderer (test-suite runs), which now also renders
variables in media URLs.
github-actions
Bot
added
documentation
⏱️ pre-commit per-hook timing
⏭️ 34 skipped (no matching files changed)
|
|
🌿 Preview your docs: https://opik-preview-01a144df-7871-770c-bad1-f9339a17cb0a.docs.buildwithfern.com/docs/opik No broken links found Unverified links (timeout / rate-limited / server error — not failing the check)• https://aistudio.google.com/apikey (401) 📌 Results for commit 89d6181 |
![baz-reviewer[bot]](https://avatars.githubusercontent.com/in/933528?s=60&v=4){kind=small}
| renderedParts.addAll(renderMediaPart(part, context)); | ||
| } |
There was a problem hiding this comment.
Video and audio list entries sent as text
Video and audio parts are sent as their JSON text because toLangChain4jMessage only handles text and image URLs; add video_url and audio_url branches that call addVideoUrl and addAudioUrl, and cover the fix with a request-level test.
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
`apps/opik-backend/src/main/java/com/comet/opik/domain/ExperimentMessageRenderer.java`
around lines 73-74, update the downstream `toLangChain4jMessage` media conversion logic
around lines 170-182 to recognize `video_url` and `audio_url` parts. Route these parts
through `OpikUserMessage.Builder.addVideoUrl` and `addAudioUrl` instead of the default
`addText(part.toString())` branch, while preserving existing text and image handling.
Add a request-level test covering list-expanded video and audio URLs.
There was a problem hiding this comment.
Commit 98e0aed addressed this comment by Added video_url and audio_url branches using addVideoUrl and addAudioUrl. A test also verifies the resulting request contains the correct media parts and URLs.
| JsonNode parsed; | ||
| try { | ||
| parsed = JsonUtils.getJsonNodeFromString(value); | ||
| } catch (UncheckedIOException e) { | ||
| return Optional.empty(); |
There was a problem hiding this comment.
Malformed media input failures disappear silently
Malformed JSON is silently treated as a URL, so invalid dataset input produces the wrong media behavior without any diagnostic. Log the parsing exception with safe variable/request context before falling back to non-list rendering, or propagate it if malformed lists are unsupported.
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
`apps/opik-backend/src/main/java/com/comet/opik/domain/ExperimentMessageRenderer.java`
around lines 121-125, update `getListFromSingleVariable` and its `renderMediaPart`
fallback handling so malformed JSON is not silently converted to `Optional.empty()`.
Catch the specific JSON parsing exception, log a safe diagnostic including the variable
name or other non-sensitive context, then apply the ordinary URL fallback only when
appropriate; otherwise propagate the parsing failure if malformed list input is
unsupported.
| processedContent = message.content.flatMap( | ||
| (part): Array<TextPart | ImagePart | VideoPart | AudioPart> => { |
There was a problem hiding this comment.
Uncapped media-list expansion inflates request size
A single dataset cell can expand into thousands of media parts, inflating the completion request and browser memory despite the 100 MB per-string and 512 MB per-request limits. Cap expanded entries or rendered bytes before flatMap, and enforce the same limit in the backend ExperimentMessageRenderer.
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
`apps/opik-frontend/src/v2/pages/PlaygroundPage/usePromptDatasetItemCombination.ts`
around lines 103-104, the `transformMessageIntoProviderMessage` media `flatMap` expands
every URL returned by `getMediaUrlsFromListVariable` without a cardinality or
payload-size limit. Add a shared, explicit cap on expanded media entries and/or total
rendered bytes before or during expansion, preserving existing rendering behavior while
preventing oversized requests. Apply the same limit and validation strategy in the
backend `ExperimentMessageRenderer` media-list expansion so frontend and backend enforce
consistent bounds.
|
🔄 Test environment deployment process has started Phase 1: Deploying base version You can monitor the progress here. |
|
✅ Test environment is now available! To configure additional Environment variables for your environment, run [Deploy Opik AdHoc Environment workflow] (https://github.com/comet-ml/comet-deployment/actions/workflows/deploy_opik_adhoc_env.yaml) Access Information
The deployment has completed successfully and the version has been verified. |
… JSON text was truncated The Playground reads dataset items from the truncated list API and only fetches the full item when it sees a truncation marker. A list stored as JSON text (as CSV uploads store it) comes back cut at 10,001 chars with its leading JSON quote, and its escaped base64 items never get the [image] placeholder, so the row was not re-fetched and the cut text was sent as one broken image URL. Found on the PR env with a JSON-text list of two images.
aadereiko
added
test-environment
|
🔄 Test environment deployment process has started Phase 1: Deploying base version You can monitor the progress here. |
|
✅ Test environment is now available! To configure additional Environment variables for your environment, run [Deploy Opik AdHoc Environment workflow] (https://github.com/comet-ml/comet-deployment/actions/workflows/deploy_opik_adhoc_env.yaml) Access Information
The deployment has completed successfully and the version has been verified. |
…os, audio or URLs was cut The truncated list API only swaps base64 images for an [image] placeholder, so a list of videos, audio or URLs longer than 10,001 chars came back as a cut string with no marker, and the Playground sent it as one broken URL. Found on the PR env with two base64 videos in one list. A string that starts like a list of strings but no longer parses as JSON is now treated as cut, which also covers lists stored as JSON text; complete lists still parse and are not re-fetched.
aadereiko
added
test-environment
|
🔄 Test environment deployment process has started Phase 1: Deploying base version You can monitor the progress here. |
| const TRUNCATED_MEDIA_REGEX = | ||
| /\[image(?:_\d+)?]|^"?data:(image|video|audio)\//i; |
There was a problem hiding this comment.
Regex constant hides its matching pattern role
Rename TRUNCATED_MEDIA_REGEX to TRUNCATED_MEDIA_PATTERN to follow the naming convention, and update its use at line 210.
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
apps/opik-frontend/src/lib/media.ts around lines 193-194, rename the
`TRUNCATED_MEDIA_REGEX` constant used by `containsTruncatedMedia` to the established
`TRUNCATED_MEDIA_PATTERN` naming convention. Update the reference around line 210 so the
truncated-media detection behavior remains unchanged.
| // A list is cut the same way, into a string that no longer parses: a real list of videos, audio or URLs | ||
| // (only base64 images get the [image] placeholder), or a list stored as JSON text, as CSV uploads store it. | ||
| const isCutList = (value: string): boolean => { | ||
| if (!/^"?\s*\[(?:\s|\\[nrt])*\\?"/.test(value)) return false; |
There was a problem hiding this comment.
Truncated non-string-first media lists skip hydration
Mixed lists with a non-string first item can render as one truncated JSON value instead of expanding because isCutList only recognizes lists whose first element is a quoted string; restore detection for both escaped and raw list prefixes so containsTruncatedMedia fetches the full dataset item.
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
`apps/opik-frontend/src/lib/media.ts` around lines 199-205, the `isCutList` detection
incorrectly requires a truncated JSON array’s first element to be a quoted string,
regressing support for inputs such as `"[{`, `"[1`, `"[null`, and raw `[{`. Refactor the
initial pattern to recognize an optional ClickHouse quote followed by any JSON array
opening, regardless of the first element, while keeping the `JSON.parse` check so valid
complete lists are not flagged. Add regression tests covering these non-string-first
truncated lists.
…te runs buildChatCompletionRequest only mapped text and image_url parts; video_url and audio_url fell through to addText(part.toString()), so the model got the part's JSON as text and invented an answer. Found on the PR env: Gemini described two room videos as "Minecraft" on the test-suite path while the same videos were described correctly in a dataset run. Uses the same OpikUserMessage builder calls as the chat-completions deserializer.
aadereiko
added
test-environment
|
🔄 Test environment deployment process has started Phase 1: Deploying base version You can monitor the progress here. |
|
✅ Test environment is now available! To configure additional Environment variables for your environment, run [Deploy Opik AdHoc Environment workflow] (https://github.com/comet-ml/comet-deployment/actions/workflows/deploy_opik_adhoc_env.yaml) Access Information
The deployment has completed successfully and the version has been verified. |
This branch has not been deployed
2 participants
Details
A user wants to send a different number of images per dataset item in one Playground prompt. Today an image, video or audio slot always becomes exactly one part, so the only option is fixed slots (
{{image_1}},{{image_2}}, ...). A list in one column doesn't work:url1,url2(how JS joins an array), and a list from a CSV upload stays["url1", ...]. The provider gets one broken URL and the row shows "Run failed: invalid image input". A 1-item list only works by accident./v1/private/experiments/execute):ExperimentMessageRendererrenders variables in text parts only, so the provider gets the literal{{images}}as the image URL. Fixed slots are broken there too: they send{{image_1}}and{{image_2}}.The change: if a media slot holds exactly one variable (
{{images}},{{ images }},{{{images}}}or{{& images}}) and its value is a list, the slot becomes one part per item, in order. An empty list sends no part. A JSON-array string counts as a list, because CSV uploads store every cell as text. Anything else (a single URL,https://cdn/{{id}}.png, an object, broken JSON) renders exactly as before.getMediaUrlsFromListVariableinlib/llm.ts;usePromptDatasetItemCombinationnow usesflatMapover the content parts, since one part can turn into several.ExperimentMessageRenderer.renderMessagesrenders variables inimage_url/video_url/audio_urlURLs and expands lists the same way. Extra fields on a part (likedetail) are kept on every copy.Two more fixes, found while testing on the PR env:
containsTruncatedMediainlib/media.ts): the Playground reads dataset rows from the list API withtruncate=true, which cuts every cell at 10,001 chars. Only base64 images that are real JSON strings get an[image]marker; a list of videos, audio or URLs, or a list stored as JSON text (as CSV uploads store it), comes back cut with no marker. So the Playground didn't load the full row and sent the cut text as one broken URL ("File URL is invalid" on GPT, an empty answer on Gemini). A string that starts like a list of strings but no longer parses as JSON now counts as cut, and the full row is loaded. A complete list still parses and is not re-fetched, and text like[link] to the docsdoesn't match.ExperimentMessageRenderer.toLangChain4jMessage, also broken on main): test-suite runs mapped onlytextandimage_urlparts.video_urlandaudio_urlparts were sent as their JSON text, and Gemini made up an answer (two room videos became "Minecraft"). They're now sent as video/audio parts.Not covered here: Gemini on Vertex AI with base64 media needs #8954. Online-evaluation judges and the Python SDK
ChatPromptdon't expand lists, and test-suite runs only see top-level columns ({{input.images}}resolves on dataset runs only, as before).Change checklist
Issues
Related: CUST_7336, OPIK_8790 (same customer thread).
AI-WATERMARK
AI-WATERMARK: yes
Testing
npm run test(FE, full suite): 272 files, 4,214 tests pass;npx tsc --noEmitand ESLint are clean. New:llm.test.ts(getMediaUrlsFromListVariable: tag variants, nested path, JSON text, non-string items, empty list, and the cases that must stay one part) andusePromptDatasetItemCombination.mediaList.test.ts(6 tests through the hook: order between text parts, CSV text, empty list, list also used in text, video/audio, fixed slots unchanged), andmedia.test.ts("cut lists": cut video/URL/JSON-text lists are detected; complete lists and[link]text are not).mvn test -Dtest=ExperimentMessageRendererTest: 57 tests, 0 failures (17 new). New tests cover the same list sources and non-list values, video/audio, the empty list, the finalChatCompletionRequest, and video/audio parts reaching it as media.url1,url2URL; literal{{images}}; cut lists not detected; video/audio sent as text).scripts/dev-runner.sh), one dataset and one test suite with 4 items: 1 image (JSON list), 2 images (JSON text, as from CSV), 3 and 5 images, plusimage_1/image_2columns for the fixed-slot case. Each image is a labelled PNG ("1 RED", "2 BLUE", ...).gemma3:4b).{{images}}{{images}}{{images}}{{images}}on every item{{images}}{{image_1}}+{{image_2}}{{image_1}}and{{image_2}}{{image_1}}+{{image_2}}2.2.103-8981-merge-3944), real models:gpt-4.1-mini(OpenAI) andgemini-3.8-flash(Google AI Studio). 12 datasets and test suites, seeded through the API and the CSV upload. For each row I checked how many media parts the trace shows, not only the answer.imageslisting_id+images{{input.images}}inside an object column{{videos}}video_url(expected){{image_1}}+{{image_2}}(old fixed slots)An empty list sends no image part on both paths. GPT then asks for images; Gemini makes some up. That's the model, not the request.
MessageContentNormalizerflattens it and skips Opik's own content type, so the text is lost too). I named the local models to match the*qwen*vl*vision pattern to get around it. It's in the ticket and needs its own fix.Screenshots
Local stack, Ollama
gemma3:4b, prompt with one image slot{{images}}. Rows hold 1 image (JSON list), 2 images (JSON text, as from CSV), 5 and 3 images.main, dataset run: only the 1-image row answers:
This PR, dataset run: every row names each image and the total:
This PR, test-suite run (the suite has no assertions, hence "Skipped"):
PR env, numbered tiles: both models list the tiles in the right order, up to 12:
PR env, a list stored as JSON text with newlines (row 1). Before the truncation fix, the cut row was sent as one broken URL:
After: the full row is loaded and both images reach the model:
PR env, two base64 videos in one
{{videos}}slot. Gemini gets both; OpenAI's API doesn't take video:Documentation
docs-v2/prompt_engineering/playground.mdx: new "Images, video, and audio" subsection under Template variables.