Skip to content

feat: Add examples field to Tool schema for enhanced LLM guidance, discovery UX and automated testing #615

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

sambhav
Copy link

@sambhav sambhav commented May 31, 2025

This PR introduces a comprehensive examples system to the MCP Tool schema that addresses three critical use cases: LLM guidance, human documentation, and automated testing. By adding concrete examples to tool definitions, we enable better tool discovery, understanding, and validation across the entire MCP ecosystem.

Motivation and Context

Organizations implementing MCP need better ways to help LLMs understand and use tools effectively. Currently, tools only have basic descriptions, which can lead to:

  • Poor tool selection by LLMs due to unclear capabilities
  • Increased hallucination when models guess at tool behavior
  • Lack of concrete examples for developers learning to implement tools
  • No standardized way to test tool implementations

This change enables 1-shot learning for LLMs, provides clear documentation for developers, and establishes a foundation for automated testing and tool registries.

How Has This Been Tested?

  • Schema validation passes with npm run validate:schema
  • JSON schema generation works correctly with npm run generate:json
  • TypeScript compilation succeeds
  • Documentation examples are comprehensive and accurate
  • Examples demonstrate all key features and edge cases including basic usage patterns, error handling, and complex scenarios with structured outputs

The examples have been tested with real tool implementations to ensure they provide meaningful guidance for both LLMs and human developers.

Breaking Changes

None. This is a purely additive change - the examples field is optional and all existing tool definitions remain valid.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation update

Checklist

  • I have read the MCP Documentation
  • My code follows the repository's style guidelines
  • New and existing tests pass locally
  • I have added appropriate error handling
  • I have added or updated documentation as needed

Additional context

Key Implementation Details

New ToolExample interface with:

  • id: Unique identifier for automated testing and LLM disambiguation
  • description: Clear explanation targeting both humans and LLMs
  • input: Example arguments matching the tool's input schema
  • output: Expected result structure with content, structured data, and error status

Enhanced Tool interface to include:

  • Optional examples array of ToolExample objects
  • Enhanced description emphasizing dual human/LLM audience

Example Usage

{
  name: "calculate_fibonacci",
  description: "Calculate the nth Fibonacci number for mathematical operations",
  inputSchema: {
    type: "object",
    properties: {
      n: { type: "integer", minimum: 0 }
    },
    required: ["n"]
  },
  examples: [
    {
      id: "basic_calculation",
      description: "Standard calculation case - compute the 5th Fibonacci number",
      input: { n: 5 },
      output: {
        content: [{ type: "text", text: "The 5th Fibonacci number is 5" }],
        structuredContent: { result: 5, sequence: [0, 1, 1, 2, 3, 5] }
      }
    }
  ]
}

Impact and Benefits

  1. Improving LLM Performance: Better tool understanding leads to more accurate and reliable AI interactions
  2. Enabling Better Tooling: Tool registries, IDEs, and documentation systems can leverage examples for richer experiences
  3. Supporting Quality Assurance: Standardized examples enable comprehensive testing and validation

This enhancement adds a comprehensive examples system to the MCP Tool schema that serves multiple critical purposes:

**LLM Guidance & 1-Shot Learning:**
- Provides concrete examples that help LLMs understand tool behavior and expected input/output formats
- Enables better tool selection by showing specific use cases and scenarios
- Reduces hallucination by giving models clear patterns to follow

**Human Documentation:**
- Offers developers clear usage examples with realistic data
- Shows edge cases and error handling patterns
- Demonstrates the relationship between inputs and outputs

**Automated Testing & Validation:**
- Enables comprehensive testing of tool functionality over time
- Supports regression testing and quality assurance
- Provides standardized test cases for tool implementations

**Tool Registry & Discovery:**
- Allows tool registries to showcase capabilities through concrete examples
- Helps users understand what tools can actually do
- Facilitates better tool discovery and adoption

**Schema Changes:**
- Added ToolExample interface with id, description, input, and output fields
- Updated Tool interface to include optional examples array
- Enhanced documentation to emphasize dual human/LLM audience
- Regenerated JSON schema with comprehensive validation

**Documentation Updates:**
- Updated tools specification with detailed examples and best practices
- Added guidance for writing effective examples
- Included comprehensive example demonstrating all features

This change positions MCP as a leader in tool discoverability and usability, making it easier for both humans and AI systems to understand and effectively use available tools.
@sambhav sambhav changed the title feat: Add examples field to Tool schema for enhanced LLM guidance and automated testing Add examples field to Tool schema for enhanced LLM guidance and automated testing May 31, 2025
@sambhav sambhav changed the title Add examples field to Tool schema for enhanced LLM guidance and automated testing feat: Add examples field to Tool schema for enhanced LLM guidance and automated testing May 31, 2025
@sambhav sambhav changed the title feat: Add examples field to Tool schema for enhanced LLM guidance and automated testing feat: Add examples field to Tool schema for enhanced LLM guidance, discovery UX automated testing May 31, 2025
@sambhav sambhav changed the title feat: Add examples field to Tool schema for enhanced LLM guidance, discovery UX automated testing feat: Add examples field to Tool schema for enhanced LLM guidance, discovery UX and automated testing May 31, 2025
@evalstate
Copy link
Contributor

I'm not currently in agreement with this PR.

This type of longform API/Prompt Engineering information may be better delivered within a normal MCP Prompt or using the Servers instructions field.

The approach also doesn't directly support ICL as examples are not shown as User/Assistant pairs. It also delegates the use of this information to the Host which is then expected to turn this in to a prompt for the LLM. Ironically, I think this additional layer will make the problem it is trying to solve worse overall as it leads to a token consuming arms race. Genuine ICL or Fine Tuning is more appropriate if the existing mechanisms (including instructions and prompts) aren't performing well enough.

I do agree that making this information available as external metadata is highly valuable, and if done so would enable the Host to consume it from there if desired. I'm just not convinced this should be core protocol at the moment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants