Skip to content

Fix: Handle missing 'choices' field in Azure OpenAI responses #13201

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Fix: Handle missing 'choices' field in Azure OpenAI responses #13201

wants to merge 3 commits into from
+171 −0

Conversation

colesmcintosh
Copy link
Collaborator

@colesmcintosh colesmcintosh commented Aug 1, 2025 •
edited
Loading

Title

Fix: Handle missing 'choices' field in Azure OpenAI responses

Relevant issues

Fixes #13139

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
  • I have added a screenshot of my new test passing locally
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible, it only solves 1 specific problem

Type

Bug Fix

Changes

This PR fixes a KeyError that occurs when Azure OpenAI returns responses without the 'choices' field during batch completion operations.

Root Cause

Azure OpenAI can return responses without the 'choices' field in certain error scenarios, such as:

  • Content filtering responses that only contain prompt_filter_results
  • Authentication/authorization failures
  • Deployment configuration errors

Implementation

Added defensive checks in convert_dict_to_response.py at three critical locations:

  • convert_to_streaming_response_async (lines 115-119)
  • convert_to_streaming_response (lines 188-192)
  • convert_to_model_response_object (lines 515-519)

The checks verify that:

  1. The 'choices' key exists in the response object
  2. The value is not None

When either condition fails, the code now raises a descriptive exception with the response content instead of a KeyError.

Testing

Comprehensive tests were created to verify all edge cases:

  • Missing 'choices' field entirely
  • 'choices' field present but with None value
  • Streaming responses without 'choices'
  • Async streaming responses without 'choices'
  • Valid responses continue to work correctly
Screenshot 2025-07-31 at 9 02 21 PM

All tests pass successfully, confirming the fix handles the intermittent failures reported in the issue.

@vercel Vercel
Copy link

vercel bot commented Aug 1, 2025 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
litellm ✅ Ready (Inspect) Visit Preview 💬 Add feedback Aug 1, 2025 8:59pm

@colesmcintosh colesmcintosh marked this pull request as draft August 1, 2025 03:01
@colesmcintosh
fix: handle missing 'choices' field in Azure OpenAI responses
07ca0cf 
- Add defensive checks before accessing response_object['choices']
- Check if 'choices' key exists and is not None
- Raise descriptive exception instead of KeyError
- Add comprehensive tests for missing choices field scenarios
- Fixes intermittent batch_completion failures with Azure OpenAI

Fixes BerriAI#13139
@colesmcintosh colesmcintosh force-pushed the fix/azure-batch-completion-keyerror branch from 11051a8 to 07ca0cf Compare August 1, 2025 03:01
@colesmcintosh colesmcintosh marked this pull request as ready for review August 1, 2025 03:02
@krrishdholakia
Copy link
Contributor

how does this fix the problem?

18:03:20 - LiteLLM:DEBUG: utils.py:349 - RAW RESPONSE:
{"id": "chatcmpl-Bz3YpCinhof67sgrd2f5z5Iri9nim", "choices": [{"finish_reason": "stop", "index": 0, "logprobs": null, "message": {"content": "The capital of Germany is Berlin.", "refusal": null, "role": "assistant", "annotations": [], "audio": null, "function_call": null, "tool_calls": null}, "content_filter_results": {"hate": {"filtered": false, "severity": "safe"}, "self_harm": {"filtered": false, "severity": "safe"}, "sexual": {"filtered": false, "severity": "safe"}, "violence": {"filtered": false, "severity": "safe"}}}], "created": 1753891395, "model": "gpt-4o-2024-08-06", "object": "chat.completion", "service_tier": null, "system_fingerprint": "fp_ee1d74bde0", "usage": {"completion_tokens": 8, "prompt_tokens": 22, "total_tokens": 30, "completion_tokens_details": {"accepted_prediction_tokens": 0, "audio_tokens": 0, "reasoning_tokens": 0, "rejected_prediction_tokens": 0}, "prompt_tokens_details": {"audio_tokens": 0, "cached_tokens": 0}}, "prompt_filter_results": [{"prompt_index": 0, "content_filter_results": {"hate": {"filtered": false, "severity": "safe"}, "self_harm": {"filtered": false, "severity": "safe"}, "sexual": {"filtered": false, "severity": "safe"}, "violence": {"filtered": false, "severity": "safe"}}}]}

18:03:20 - LiteLLM:INFO: utils.py:1260 - Wrapper: Completed Call, calling success_handler
18:03:20 - LiteLLM:DEBUG: cost_calculator.py:675 - selected model name for cost calculation: azure/gpt-4o-2024-08-06
18:03:20 - LiteLLM:DEBUG: utils.py:4620 - checking potential_model_names in litellm.model_cost: {'split_model': 'gpt-4o-2024-08-06', 'combined_model_name': 'azure/gpt-4o-2024-08-06', 'stripped_model_name': 'gpt-4o-2024-08-06', 'combined_stripped_model_name': 'azure/gpt-4o-2024-08-06', 'custom_llm_provider': 'azure'}
18:03:20 - LiteLLM:DEBUG: utils.py:4925 - model_info: {'key': 'azure/gpt-4o-2024-08-06', 'max_tokens': 16384, 'max_input_tokens': 128000, 'max_output_tokens': 16384, 'input_cost_per_token': 2.5e-06, 'cache_creation_input_token_cost': None, 'cache_read_input_token_cost': 1.25e-06, 'input_cost_per_character': None, 'input_cost_per_token_above_128k_tokens': None, 'input_cost_per_token_above_200k_tokens': None, 'input_cost_per_query': None, 'input_cost_per_second': None, 'input_cost_per_audio_token': None, 'input_cost_per_token_batches': None, 'output_cost_per_token_batches': None, 'output_cost_per_token': 1e-05, 'output_cost_per_audio_token': None, 'output_cost_per_character': None, 'output_cost_per_reasoning_token': None, 'output_cost_per_token_above_128k_tokens': None, 'output_cost_per_character_above_128k_tokens': None, 'output_cost_per_token_above_200k_tokens': None, 'output_cost_per_second': None, 'output_cost_per_image': None, 'output_vector_size': None, 'citation_cost_per_token': None, 'litellm_provider': 'azure', 'mode': 'chat', 'supports_system_messages': None, 'supports_response_schema': True, 'supports_vision': True, 'supports_function_calling': True, 'supports_tool_choice': True, 'supports_assistant_prefill': None, 'supports_prompt_caching': True, 'supports_audio_input': None, 'supports_audio_output': None, 'supports_pdf_input': None, 'supports_embedding_image_input': None, 'supports_native_streaming': None, 'supports_web_search': None, 'supports_url_context': None, 'supports_reasoning': None, 'supports_computer_use': None, 'search_context_cost_per_query': None, 'tpm': None, 'rpm': None}
18:03:20 - LiteLLM:DEBUG: utils.py:4620 - checking potential_model_names in litellm.model_cost: {'split_model': 'gpt-4o-2024-08-06', 'combined_model_name': 'azure/gpt-4o-2024-08-06', 'stripped_model_name': 'gpt-4o-2024-08-06', 'combined_stripped_model_name': 'azure/gpt-4o-2024-08-06', 'custom_llm_provider': 'azure'}
litellm.APIError: AzureException APIError - Invalid response object Traceback (most recent call last):
File "/Users/mbn/Library/Caches/pypoetry/virtualenvs/llm-client-a7HKjXJA-py3.12/lib/python3.12/site-packages/litellm/litellm_core_utils/llm_response_utils/convert_dict_to_response.py", line 502, in convert_to_model_response_object
assert response_object["choices"] is not None and isinstance(
~~~~~~~~~~~~~~~^^^^^^^^^^^
KeyError: 'choices'

looking at the raw response, i can see choices exists

@ellonde
Copy link

ellonde commented Aug 1, 2025

I reported the issue, and I can confirm that the choices was indeed present in the response when failing.
In the case if this specific error, I tracked it down to when response.model_dump() was called it didn't parse the whole response. But like I state in the bug report, this was just one error I got, sometimes. So I think the underlying issue is something else.

@colesmcintosh colesmcintosh marked this pull request as draft August 1, 2025 21:17
@krrishdholakia
Copy link
Contributor

when will this be ready for review?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug]: batch_completion not working, sometimes?
3 participants
@colesmcintosh @krrishdholakia @ellonde