Skip to content

Retry mechanism for ModelBehaviorError #325

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
ProblemFactory opened this issue Mar 25, 2025 · 10 comments
Open

Retry mechanism for ModelBehaviorError #325

ProblemFactory opened this issue Mar 25, 2025 · 10 comments
Labels
enhancement New feature or request

Comments

@ProblemFactory
Copy link

ProblemFactory commented Mar 25, 2025 •
edited
Loading

I have extensively use this framework during the recent week. It performs pretty well, except on rare circumstances, the LLM will attempt to call a nonexistent tool, which crashed a whole 10-min agent run. Can you implement a retry mechanism allowing re-execute the errored LLM call to have a chance to recover from the it?
@ProblemFactory ProblemFactory added the enhancement New feature or request label Mar 25, 2025
@rm-openai
Copy link
Collaborator

Hmm can you share some code? function_tool already catches errors and attempts to have the LLM run again. Would be useful to know where it crashed so that I can debug.

@ProblemFactory
Copy link
Author

ProblemFactory commented Mar 25, 2025 •
edited
Loading

Hmm can you share some code? function_tool already catches errors and attempts to have the LLM run again. Would be useful to know where it crashed so that I can debug.

This is the exception occurred

Traceback (most recent call last):
  File "/app/python/deep_search/agents/executor.py", line 81, in run_main_agent
    async for event in resp.stream_events():
  File "/home/work/.local/lib/python3.11/site-packages/agents/result.py", line 186, in stream_events
    raise self._stored_exception
  File "/home/work/.local/lib/python3.11/site-packages/agents/run.py", line 537, in _run_streamed_impl
    turn_result = await cls._run_single_turn_streamed(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/work/.local/lib/python3.11/site-packages/agents/run.py", line 674, in _run_single_turn_streamed
    single_step_result = await cls._get_single_step_result_from_response(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/work/.local/lib/python3.11/site-packages/agents/run.py", line 755, in _get_single_step_result_from_response
    processed_response = RunImpl.process_model_response(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/work/.local/lib/python3.11/site-packages/agents/_run_impl.py", line 367, in process_model_response
    raise ModelBehaviorError(f"Tool {output.name} not found in agent {agent.name}")
agents.exceptions.ModelBehaviorError: Tool search_linkedin not found in agent DeepSearch

The agent is defined as

main_agent = Agent(
        "DeepSearch", 
        MAIN_AGENT_PROMPT, 
        tools=[
            search_google_news, 
            search_google,
            visit_page,
            get_linkedin_job_postings
        ]
    )

Run config

run_config = RunConfig(
        model=DEFAULT_MODEL, #bedrock claude-3.7-sonnet
        model_provider=OpenAIProvider(
            api_key=MODEL_API_KEY,
            base_url=MODEL_BASE_URL #litellm proxy URL
        )

I ran the agent with

Runner.run_streamed(
                main_agent, 
                self.user_query, 
                max_turns=50, 
                run_config=run_config
            )

@rm-openai
Copy link
Collaborator

Ah got it. I can add something to fix this.

@tim5ly
Copy link

tim5ly commented Mar 26, 2025 •
edited
Loading

@rm-openai hey! having the same question regarding retries and also want to know the proper way of using tenacity.retry with Runner.run_streamed (if possible).

Thanks!

@jackien1
Copy link

jackien1 commented Apr 8, 2025

@rm-openai also curious about retries with Runner.run_streamed or what is the proper mechanism.

@leoxs22
Copy link

leoxs22 commented Apr 28, 2025

Also for rate-limit. I have a flow that might make 50 tool calls and I'm hitting the tokens per minute limit. I would expect it to continue with an exponential retry

@exiao
Copy link

exiao commented May 2, 2025

Need this one also

@dilwong
Copy link

dilwong commented May 8, 2025

This would be useful. Using an AzureOpenAI model that supports Structured Outputs (specifically gpt4o-2024-11-20), I've occasionally seen ModelBehaviorError: Invalid JSON when parsing followed by two copies of the structured output response that I specified in output_type. Does anyone know why this happens? It's difficult to reproduce because of the stocastic nature of these LLMs.

@LatVAlY
Copy link

LatVAlY commented May 8, 2025 •
edited
Loading

@dilwong not related but, I have issue getting AzureOpenAI with Agents to run, I always get the api-version issue 2025-01-01-preview or 404, yet when I try Azure OpenAi alone it works, once I set it with the agent i get the issue
I would like to know how did you set it up.
Thanks!

@dilwong
Copy link

dilwong commented May 8, 2025 •
edited
Loading

@dilwong not related but, I have issue getting AzureOpenAI with Agents to run, I always get the api-version issue 2025-01-01-preview or 404, yet when I try Azure OpenAi alone it works, once I set it with the agent i get the issue I would like to know how did you set it up. Thanks!

@LatVAlY Probably not the place for this, but here is a minimal example:

from agents import Agent, Runner, OpenAIChatCompletionsModel
from openai import AsyncAzureOpenAI
import os


client = AsyncAzureOpenAI(
   api_key=os.getenv('AZURE_OPENAI_API_KEY'),
   api_version=os.getenv('OPENAI_API_VERSION'), # My API version is '2024-12-01-preview'
   azure_endpoint=os.getenv('AZURE_OPENAI_ENDPOINT')
)


MODEL_NAME = "my-company's-deployment-name-gpt4omini-2024-07-18"


agent = Agent(name=AGENT_NAME,
               instructions=INSTRUCTIONS,
               model=OpenAIChatCompletionsModel(model=MODEL_NAME, openai_client=client),
               tools=[WHATEVER_TOOLS],
               output_type=PYDANTIC_BASE_MODEL
           )


result = Runner.run_sync(agent, QUERY)
print(result.final_output)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@exiao @ProblemFactory @leoxs22 @jackien1 @dilwong @LatVAlY @tim5ly @rm-openai