What does "Key '$defs' is not supported in schema, ignoring" mean?

[pinkponk]

I keep getting Key '$defs' is not supported in schema, ignoring when I use gemini flash 1.5 with structured output but I can't seem to understand what the warning means. Cannot find anything on the web. Anyone knows? I think it might have something to do with nested pydantic models

I use the langchain-google-vertexai pkg at version 2.0.9

[RobSisson]

I've hit the same issue when doing the Langgraph Academy course in module 5 - I believe it occurs for me when using a nested schema in Trustcall with gemini-1.5-flash.

Not sure if this is gemini / vertex ai related, as I've hit some other bugs throughout the course when running the provided notebooks without changing anything other than the model.

One which I'll highlight is another one which I've not been able to solve and also relates to using Trustcall.

Error: 'Key 'examples' is not supported in schema, ignoring'

Circumstance: Using gemini-1.5-flash to update an existing collection using Trustcall

# Invoke the extractor with our updated conversation and existing memories result = trustcall_extractor.invoke({"messages": updated_conversation, "existing": existing_memories})

Schema:
`from pydantic import BaseModel, Field

class Memory(BaseModel):
content: str = Field(description="The main content of the memory. For example: User expressed interest in learning about French.")

class MemoryCollection(BaseModel):
memories: list[Memory] = Field(description="A list of memories about the user.")`

Unsolved: I think it is relating to gemini struggling to respond when the with_structured_output function, and it continuously generates the "example" field value which trustcall can't deal with.

I've tried changing the trustcall prompt and putting the temperature to 0, both with no luck.

[RobSisson]

As a note - been playing around and all of the errors I faced working with Gemini were solved completely when using OpenAI instead.

[pinkponk]

Yes running the openai models works much better with the structured output. I however would like to know what the warning actually means when using the gemini models

[agercas]

The issue is indeed with the nested pydantic model. Google/VertexAI doesn't support pydantic BaseModel directly and it is not even clear based on the documentation if they support complex nested types with structured output

https://ai.google.dev/gemini-api/docs/structured-output?lang=python

If you use pydantic BaseModel, this package converts it to a custom google schema using one of these functions

langchain-google/libs/vertexai/langchain_google_vertexai/functions_utils.py

Line 148 in 1e3af5a

logger.warning(f"Key '{key}' is not supported in schema, ignoring")

langchain-google/libs/genai/langchain_google_genai/_function_utils.py

Line 117 in 1e3af5a

logger.warning(f"Key '{key}' is not supported in schema, ignoring")

On the other hand, OpenAI documentation is pretty clear on using pydantic for nested definitions
https://platform.openai.com/docs/guides/structured-outputs

Code example

from pydantic import BaseModel
from typing import List
import json

class User(BaseModel):
    name: str

class Organization(BaseModel):
    name: str
    users: List[User]

# Example of creating an instance
user1 = User(name="John Doe")
user2 = User(name="Jane Smith")

org = Organization(
    name="Some Org",
    users=[user1, user2]
)

print(json.dumps(org.model_json_schema(), indent=2))


from langchain_google_vertexai.functions_utils import _format_json_schema_to_gapic
print()
print(json.dumps(_format_json_schema_to_gapic(org.schema()), indent=2))

above code outputs

WARNING:langchain_google_vertexai.functions_utils:Key '$defs' is not supported in schema, ignoring
WARNING:langchain_google_vertexai.functions_utils:Key '$ref' is not supported in schema, ignoring
{
  "$defs": {
    "User": {
      "properties": {
        "name": {
          "title": "Name",
          "type": "string"
        }
      },
      "required": [
        "name"
      ],
      "title": "User",
      "type": "object"
    }
  },
  "properties": {
    "name": {
      "title": "Name",
      "type": "string"
    },
    "users": {
      "items": {
        "$ref": "#/$defs/User"
      },
      "title": "Users",
      "type": "array"
    }
  },
  "required": [
    "name",
    "users"
  ],
  "title": "Organization",
  "type": "object"
}

{
  "properties": {
    "name": {
      "title": "Name",
      "type": "STRING"
    },
    "users": {
      "items": {},
      "title": "Users",
      "type": "ARRAY"
    }
  },
  "required": [
    "name",
    "users"
  ],
  "title": "Organization",
  "type": "OBJECT"
}