Description of the bug:

I’m encountering an issue with the gemini-1.5-flash model from the google.generativeai package when trying to generate content with a response MIME type set to application/json. According to the documentation, this MIME type should be supported, but I’m receiving the following error:

google.api_core.exceptions.InvalidArgument: 400 Function calling with a response mime type: 'application/json' is unsupported

Note: The reason I am supplying schema as text in prompt is coz flash does not support it.

Code Example

from google.generativeai import GenerativeModel

def get_products():
    return {
        "product_name": "laptop",
        "price": 20000,
        "permalink": "#link",
        "image_link": "#image"
    }

model = GenerativeModel('gemini-1.5-flash-latest',
                        tools=[get_products],
                        generation_config={"response_mime_type": "application/json"})

system_instruction = """
You are a helpful e-commerce assistant. When responding, use the following formats:
1. Normal Response: {"type": "message", "data": str}
2. Product Response: {"type": "products", "data": list[{"product_name": str, "price": int, "permalink": str, "image_link": str}]}
"""

model.generate_content(system_instruction)

Use Case:

I am working on an e-commerce assistant that needs to generate responses in specific formats. Here’s a brief overview of my use case:

System Instruction:

I want to set up the assistant with a system instruction that defines how it should format responses. The assistant should handle two types of responses:

Normal Responses: Simple text responses with a message type.

Product Responses: Structured JSON responses with a list of products.

Normal Response:

{
  "type": "message",
  "data": "How can I help you?"
}

Products Response:

{
  "type": "products",
  "data": [
    {
      "product_name": "samsung",
      "price": 20000,
      "permalink": "#link1",
      "image_link": "#image1"
    },
    {
      "product_name": "iphone",
      "price": 100000,
      "permalink": "#link2",
      "image_link": "#image2"
    }
  ]
}

Questions

• Is there an alternative way to format responses in JSON using the gemini-1.5-flash/pro model?
• Are there any updates or workarounds for handling structured JSON responses with this model?

Thank you for your assistance!