Hi,
We have a very similar use case where we have PDF file analysis function calls that want to return:

• file name
• base64 file content
• type = input_file
• additional str information about the file

Can you expand this PR to also support the concept of a @file_function_tool?

I appreciate very much that you've done this work.
Perhaps I don't understand, but it kind of looks like all an image tool can return is one image. Is that right?
How about a tool that returns complex json output, some properties of which are images?
I have some ideas on how to implement that and am considering it.
LMK, if this method actually supports that already.

@stevemadere At that point, instead of a tool, we just have an agent participating in the conversation by posting a message made of multiple parts.

I thought agent responses have to be all string ? How does the agent response return File content types without a PR like this one?

@diwu-sf OpenAI is now promoting Responses API instead of ChatCompletion API. This new spec allows agents or models to return output as various parts of different types:

@nileshtrivedi nope, function call responses from the tool itself still must be string:
https://platform.openai.com/docs/guides/function-calling?api-mode=responses#formatting-results

That's why this PR is generating a user message to embed the function call's image output:

    @classmethod
    def image_function_tool_call_output_item(
        cls, tool_call: ResponseFunctionToolCall, output: str
    ) -> FunctionCallOutput:
        """Creates a tool call output item from a tool call and its output."""
        return [
            {
                "call_id": tool_call.call_id,
                "output": "Image generating tool is called.",
                "type": "function_call_output",
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "input_image",
                        "image_url": output,
                    }
                ],
            },
        ]

Something similar can be done for arbitrary PDF / file uploads

@nileshtrivedi:

Consider the following situation (which I suspect is about to become super common):

A MCP server to conduct web browsing operations such as navigateToUrl, takeAction. (e.g. via stagehand).

Now, such an action would need to return all of these:

1. requested information from the DOM
2. the current location (url)
3. a screenshot of the browser's screen (typically a .png) after navigating or taking the desired action.

It could store the screenshot at a publicly accessible location on the web (e.g. in a S3 bucket served via cloudfront) so that the screenshot could be returned as an http URL easily enough. (just perfect for providing in a input_image message hoisted from the MCP function call results)

The LLM at OpenAI can examine the screenshot and decide which action to take next and make a tool call to take that action.
It will need to see the new screenshot as well as the new location and perhaps some information from the DOM via stagehand's observe method.

Did I do a better job of describing the multi-modal results and why an MCP tool call would need to propagate them all simultaneously to the calling model?