Stream content and IFM tool calls in multi-format parser

[aaryamonvikram]

Summary

• Add a streaming fallback for non-delegated MultiFormatToolParser formats.
• Stream ordinary text as content when no tool marker is active.
• Buffer partial tool-call markers and emit completed IFM/tool blocks as OpenAI-compatible tool_calls deltas.
• Populate parser tool-call state so streaming responses can finish with finish_reason: tool_calls.

Context

When tools + tool_choice:auto is used, vLLM routes output through the tool parser after reasoning ends. The K2/IFM parser previously returned None in streaming mode for non-delegated formats, which caused final answer text and IFM tool calls to be suppressed.

Validation

• Ran a lightweight parser behavior check inside the patched vLLM Enroot image with this branch mounted as source:
  • normal post-reasoning text -> DeltaMessage(content=...)
  • split IFM write_file block -> OpenAI tool_calls delta with JSON arguments
  • parser state populated for tool-call finish handling

[hanseungwook]

@aaryamonvikram Created some test cases that fail and should be addressed :)

Assuming for example the tool calls are like the following

CALL_1 = <ifm|tool_call>get_weather<ifm|arg_key>city</ifm|arg_key><ifm|arg_value>Tokyo</ifm|arg_value></ifm|tool_call>

CALL_2 = <ifm|tool_call>get_time<ifm|arg_key>city</ifm|arg_key><ifm|arg_value>Seoul</ifm|arg_value></ifm|tool_call>

1. Leading newline causes delayed parsing

Incoming tool-parser delta:

"\n" + CALL_1

Expected:

content="\n"
tool_calls=[get_weather({"city": "Tokyo"})]

Actual:

content="\n"
tool_calls=[]

CALL_1 remains buffered. It is lost if another delta does not arrive.

2. Second tool call is discarded

Incoming deltas:

Delta 1: "<ifm|tool_calls>" + CALL_1
Delta 2: CALL_2 + "</ifm|tool_calls>"

Expected:

Delta 1: tool_calls[0]=get_weather(...)
Delta 2: tool_calls[1]=get_time(...)

Actual:

Delta 1: tool_calls[0]=get_weather(...)
Delta 2: None

The parser stops emitting after its first parsed tool call.

3. Final tool call is rewritten incorrectly

Final tool-parser delta:

CALL_1

Expected:

id=
name="get_weather"
arguments='{"city": "Tokyo"}'

Actual after finish processing:

id=None
name=None
arguments='"{\"city\": \"Tokyo\"}"'

The name and ID are lost, and the arguments are JSON-encoded twice.

4. Non-K2 marker incorrectly activates tool parsing

Incoming K2-parser content:

"Use <tool_call> literally in the documentation."

Expected:

content="Use <tool_call> literally in the documentation."

Actual:

content="Use "

The remaining text is buffered and may never be emitted. PR #10 incorrectly treats generic <tool_call> as a K2 marker.

5. Reasoning end and K2 tool call in the same final delta

Incoming final engine delta:

"</ifm|think>\n" + CALL_1

Expected accumulated response:

reasoning=""
content="\n"
tool_calls=[get_weather({"city": "Tokyo"})]
finish_reason="tool_calls"

Actual:

reasoning=""
content=None
tool_calls=[]
finish_reason="stop"

The reasoning parser finds the post-reasoning text, but serving finishes without invoking the tool parser. Both the newline and tool call are lost.

[aaryamonvikram]

Addressed the comments, @hanseungwook please have a look

[hanseungwook]

Combined with #12 so closing