Summary

This PR implements Voice Pipeline Orchestration for OpenAI's Realtime API, providing a comprehensive framework for managing voice interactions with the gpt-realtime model.

Motivation

While working with the OpenAI Agents SDK, I wanted to contribute a voice pipeline orchestration feature that makes it easier to build voice-enabled applications using OpenAI's Realtime API with gpt-realtime, Whisper STT, and the new Marin/Cedar voices.

What's Included

Core Implementation (packages/agents-realtime/src/voicePipeline.ts)

• VoicePipeline class with event-driven architecture
• Integration with gpt-realtime model
• Support for Marin and Cedar realtime voices
• Whisper STT integration
• WebRTC support for ultra-low latency (<100ms)
• Voice Activity Detection (VAD)
• Audio enhancement (echo/noise suppression, gain control)
• Plugin pattern for easy RealtimeSession integration

Comprehensive Tests (packages/agents-realtime/test/voicePipeline.test.ts)

• Test coverage for all features
• Audio processing and synthesis tests
• WebRTC integration tests
• Error handling scenarios

Documentation (docs/src/content/docs/guides/voice-pipeline.mdx)

• Complete usage guide
• Configuration examples
• Best practices
• Integration with RealtimeSession

Working Example (examples/voice-pipeline/)

• Full implementation example
• Demonstrates all features
• Ready-to-run with README

Key Features

✅ OpenAI Realtime API Integration

• gpt-realtime model support
• Marin and Cedar voice options
• Whisper STT for transcription

✅ Real-time Processing

• Low-latency audio streaming
• WebRTC support for <100ms latency
• Automatic buffering
• Streaming responses

✅ Voice Activity Detection

• Configurable thresholds
• Debouncing support
• Silence detection

✅ Audio Enhancement

• Echo suppression
• Noise reduction
• Automatic gain control

✅ Developer Experience

• TypeScript with full type safety
• Event-driven API
• Plugin pattern for RealtimeSession
• Comprehensive metrics monitoring

Usage Example

import { createVoicePipeline } from '@openai/agents/realtime';

const pipeline = createVoicePipeline({
  model: 'gpt-realtime',
  voice: 'marin', // or 'cedar'
  stt: { model: 'whisper-1' }
});

pipeline.on('speech.final', (text) => {
  console.log('User said:', text);
});

await pipeline.processAudio(audioBuffer);

Testing

All tests pass with the existing test suite. The new tests follow the same patterns as existing SDK tests.

Breaking Changes

None. This is a purely additive feature that doesn't modify any existing APIs.

Checklist

• Code follows SDK patterns and conventions
• Tests written and passing
• Documentation added
• Example implementation provided
• No breaking changes
• TypeScript types included
• Follows existing code style

Notes

This contribution provides a framework for voice pipeline orchestration that integrates with OpenAI's Realtime API. The implementation focuses on providing a clean abstraction over the complexity of audio streaming, transcription, and synthesis while maintaining low latency for real-time voice interactions.

Thank you for considering this contribution! 🙏