Skip to content

Realtime docs #1153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jul 16, 2025
Merged

Realtime docs #1153

merged 1 commit into from
Jul 16, 2025
+557 −0

Conversation

rm-openai
Copy link
Collaborator

@rm-openai rm-openai commented Jul 16, 2025 •
edited
Loading

Documentation (both written and code)

@sibblegp
Copy link

@rm-openai I'm telling you, tool calls are not working properly. I've tried everything.

sibblegp
sibblegp reviewed Jul 16, 2025
View reviewed changes
docs/realtime/guide.md
handoffs=[
realtime_handoff(billing_agent, tool_description="Transfer to billing support"),
realtime_handoff(technical_agent, tool_description="Transfer to technical support"),
]
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the difference between just putting in the agent and using "realtime_handoff"? In your web demo you just have "handoffs=[agent]"?

@sibblegp
Copy link

I set up this example and it doesn't reply with the string. Even basic tool calling doesn't work yet. If I just need to be patient, let me know. It says "I'll put that up" and I can see the tool being called, but it never responds with the output.

@function_tool
def get_password() -> str:
    """
    Get the password for the user.
    """
    return "Your Password is Not Set"

AUDIO_AGENT = RealtimeAgent(
    name="ODAI-Voice",
    instructions='You are a helpful assistant. You are able to answer questions and help with tasks.',
    tools=[get_password]
)```

@rm-openai
Copy link
Collaborator Author

@sibblegp can you create a separate issue with a script to reproduce and I can take a look?

@sibblegp
Copy link

@rm-openai

Will do! It could be you are running new code. If after you publish v2.1 it's still not working, I'll make an issue.

sibblegp
sibblegp reviewed Jul 16, 2025
View reviewed changes
docs/realtime/guide.md
3. **Start the session** using `await runner.run()` which returns a RealtimeSession.
4. **Send audio or text messages** to the session using `send_audio()` or `send_message()`
5. **Listen for events** by iterating over the session - events include audio output, transcripts, tool calls, handoffs, and errors
6. **Handle interruptions** when users speak over the agent, which automatically stops current audio generation
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rm-openai Can you give an example of how you handle interruptions? I don't see any in the demo files made so far. I think you just need to detect when audio starts and then call truncate but I don't see how to do that. Thanks!

@sibblegp
Copy link

Also, neither demo runs for me. I get the same error on both the UI and the non UI demos.

Raw model event: RealtimeModelExceptionEvent(exception=1 validation...
Audio capture error: sent 1000 (OK); then received 1000 (OK)
Traceback (most recent call last):
  File "/Users/gsibble/apps/realtime-agents/examples/realtime/no_ui_demo.py", line 250, in <module>
    asyncio.run(demo.run())
    ~~~~~~~~~~~^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.13/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ~~~~~~~~~~^^^^^^
  File "/opt/homebrew/Cellar/python@3.13/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/opt/homebrew/Cellar/python@3.13/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/base_events.py", line 721, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "/Users/gsibble/apps/realtime-agents/examples/realtime/no_ui_demo.py", line 137, in run
    async for event in session:
        await self._on_event(event)
  File "/Users/gsibble/apps/openai-agents-python/examples/realtime/env_demo/lib/python3.13/site-packages/agents/realtime/session.py", line 147, in __aiter__
    raise self._stored_exception
  File "/Users/gsibble/apps/openai-agents-python/examples/realtime/env_demo/lib/python3.13/site-packages/agents/realtime/openai_realtime.py", line 169, in _listen_for_messages
    await self._handle_ws_event(parsed)
  File "/Users/gsibble/apps/openai-agents-python/examples/realtime/env_demo/lib/python3.13/site-packages/agents/realtime/openai_realtime.py", line 502, in _handle_ws_event
    await self._handle_output_item(parsed.item)
  File "/Users/gsibble/apps/openai-agents-python/examples/realtime/env_demo/lib/python3.13/site-packages/agents/realtime/openai_realtime.py", line 376, in _handle_output_item
    message_item: RealtimeMessageItem = TypeAdapter(RealtimeMessageItem).validate_python(
                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        {
        ^
    ...<5 lines>...
        }
        ^
    )
    ^
  File "/Users/gsibble/apps/openai-agents-python/examples/realtime/env_demo/lib/python3.13/site-packages/pydantic/type_adapter.py", line 421, in validate_python
    return self.validator.validate_python(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        object,
        ^^^^^^^
    ...<5 lines>...
        by_name=by_name,
        ^^^^^^^^^^^^^^^^
    )
    ^
pydantic_core._pydantic_core.ValidationError: 1 validation error for tagged-union[SystemMessageItem,UserMessageItem,AssistantMessageItem]
assistant.content.0
  Input should be a valid dictionary or instance of AssistantText [type=model_type, input_value=ConversationItemContent(i...me check', type='audio'), input_type=ConversationItemContent]
    For further information visit https://errors.pydantic.dev/2.11/v/model_type

@rm-openai rm-openai merged commit bbbcdae into main Jul 16, 2025
10 checks passed
@rm-openai rm-openai deleted the rm/pr1153 branch July 16, 2025 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sibblegp sibblegp sibblegp left review comments

@jhills20 jhills20 jhills20 approved these changes

@seratch seratch Awaiting requested review from seratch

@dkundel-openai dkundel-openai Awaiting requested review from dkundel-openai

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@rm-openai @sibblegp @jhills20