fix(eval): use function calling for tool/input mocking so non-OpenAI models work (AE-1646)#1692
Open
Chibionos wants to merge 5 commits into
Open
fix(eval): use function calling for tool/input mocking so non-OpenAI models work (AE-1646)#1692Chibionos wants to merge 5 commits into
Chibionos wants to merge 5 commits into
Conversation
…models work Tool simulation and input generation in Studio Debug and Evaluation Set runs failed with AGENT_RUNTIME.UNEXPECTED_ERROR for non-OpenAI models (Anthropic Claude via Bedrock, Gemini). The mockers requested structured output via OpenAI-only `response_format` json_schema and parsed `choices[0].message.content`; for Claude that content is empty/None, so `json.loads(...)` raised. Switch both mockers to provider-agnostic function calling (mirrors llm_as_judge_evaluator): build a forced tool that wraps the output/input schema under a `response` property, force it via tool_choice, and read `tool_calls[0].arguments["response"]` (already a parsed dict). Hoist nested `$defs` to the tool-parameters root so `$ref`s from nested Pydantic models still resolve. The normalized LLM gateway now accepts raw-dict tools so arbitrary nested schemas survive (the ToolDefinition converter only emits flat properties). Regression introduced by #1555, which started routing the agent's model into simulations; before that, simulation always used a fixed OpenAI model, so non-OpenAI providers were never exercised on this path. Fixes AE-1646. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The mocker fix in uipath depends on the dict-tool passthrough in uipath-platform, so uipath's lower-bound pin is raised to 0.1.60. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
test:uipath-langchain
…ation The normalized gateway accepts $ref/$defs in response_format but not inside a tool's parameters. Tool outputs typed as nested Pydantic models/enums (e.g. calculator's get_random_operator -> Wrapper[Operator]) produced a tool schema with $ref/$defs that the gateway rejected, so simulation failed. Inline the definitions into a self-contained schema (cyclic refs keep their $defs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
🚨 Heads up:
|
All-tool-calling regressed OpenAI tool simulation (calculator-evals 'Test Random Addition Using LLM' became flaky: gpt_4_1_mini returned wrong/empty values for a nested-enum output schema via function calling, where response_format was reliable). Make structured-output generation adaptive: prefer response_format (honored reliably by OpenAI, native $defs support) and fall back to a forced tool call only when content comes back empty (the non-OpenAI failure mode, e.g. Claude/Bedrock). Shared in generate_structured_output(), used by both mockers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Chibionos
force-pushed
the
fix/ae-1646-mocker-non-openai-models
branch
from
5e4bdb2 to
ae78cbe
Compare
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Contributor
Author
|
Re: the Full suite is green: all eval test-cases (calculator-evals, simulation-testcase, tools-evals, etc.) pass across alpha/cloud/staging; lint, SonarCloud, build all pass. |
mjnovice
reviewed
Jun 3, 2026
| logger = logging.getLogger(__name__) | ||
|
|
||
|
|
||
| def _inline_defs( |
Contributor
There was a problem hiding this comment.
Can we have separate classes for how we are doing Anthropic, OpenAI, Gemini etc. ?
mjnovice
approved these changes
Jun 3, 2026
Contributor
mjnovice
left a comment
mjnovice
left a comment
There was a problem hiding this comment.
Minor comment about making the generate_structured_output more modular.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tool simulation and input generation in Studio Debug and Evaluation Set runs failed with
AGENT_RUNTIME.UNEXPECTED_ERRORfor non-OpenAI models (Anthropic Claude via Bedrock, Gemini), but worked with OpenAI/GPT.Both eval mockers requested structured output via OpenAI-only
response_formatjson_schema and parsedresponse.choices[0].message.content. On the normalized LLM Gateway,response_formatstructured output is only honored for OpenAI models; for Claude the content comes back empty/None, sojson.loads(None)raised → wrapped asUiPathMockResponseGenerationError→AGENT_RUNTIME.UNEXPECTED_ERROR.Fixes AE-1646 (customer: Sarasota Memorial Health Care System).
Root cause / regression
Regression from #1555, which started routing the agent's model into simulations. Before that, simulation always used a fixed OpenAI model (
gpt_4_1_mini), so non-OpenAI providers were never exercised on this path — which is why Claude "worked before."Fix
Switch both mockers to provider-agnostic function calling, mirroring
llm_as_judge_evaluator(whose docstring already states function calling is the way to get structured output across OpenAI/Claude/Gemini):responseproperty, force it viatool_choice=required, and readtool_calls[0].arguments["response"](already a parsed dict).$defsto the tool-parameters root so$refs from nested Pydantic models still resolve once the schema is wrapped.chat_completionsnow accepts raw-dict tools (pass-through) so arbitrary nested schemas survive — theToolDefinitionconverter only emits flat properties.New shared helper
eval/mocks/_structured_output.pykeeps both mockers DRY.Tests
test_llm_mockable_structured_output_via_tool_call— parametrized overgpt-4.1-mini,anthropic.claude-sonnet-4-5,gemini-2.5-pro; reproduces AE-1646 (content=None + tool_calls) and asserts the new contract.test_build_response_tool_hoists_defs_to_root+ helper error-branch unit tests.test_raw_dict_tool_passthrough_mocked(platform) — asserts a nested array schema is forwarded byte-for-byte.tests/cli/evalsuite + platform mocked LLM tests green; ruff + mypy clean.Note for reviewers
The OpenAI path also moves to function-calling (no longer
response_format), matching the judge. Worth a live check that a Claude/Bedrock agent simulating a nested-model tool output round-trips through the gateway, since nested$defsin toolparametershas no prior precedent in this repo (the judge only used flat schemas).🤖 Generated with Claude Code