Module: OpenAI::Models::Realtime::RealtimeSession::TurnDetection

Extended by:
Internal::Type::Union
Defined in:
lib/openai/models/realtime/realtime_session.rb,
sig/openai/models/realtime/realtime_session.rbs

Overview

Configuration for turn detection, ether Server VAD or Semantic VAD. This can be set to null to turn off, in which case the client must manually trigger model response.

Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.

Semantic VAD is more advanced and uses a turn detection model (in conjunction with VAD) to semantically estimate whether the user has finished speaking, then dynamically sets a timeout based on this probability. For example, if user audio trails off with "uhhm", the model will score a low probability of turn end and wait longer for the user to continue speaking. This can be useful for more natural conversations, but may have a higher latency.

For gpt-realtime-whisper transcription sessions, turn detection must be set to null; VAD is not supported.

See Also:

  • OpenAI::Models::Realtime::RealtimeSession#turn_detection

Defined Under Namespace

Classes: SemanticVad, ServerVad

Instance Method Summary collapse

Methods included from Internal::Type::Union

==, ===, coerce, derefed_variants, dump, hash, inspect, to_sorbet_type, variants, variants

Methods included from Internal::Util::SorbetRuntimeSupport

#const_missing, #define_sorbet_constant!, #sorbet_constant_defined?, #to_sorbet_type, to_sorbet_type

Methods included from Internal::Type::Converter

#coerce, coerce, coerce_with_error, dump, #dump, #inspect, inspect, meta_info, new_coerce_state, type_info

Instance Method Details

#initialize(create_response: nil, idle_timeout_ms: nil, interrupt_response: nil, prefix_padding_ms: nil, silence_duration_ms: nil, threshold: nil, type: :server_vad) ⇒ Object

Server-side voice activity detection (VAD) which flips on when user speech is detected and off after a period of silence.

Parameters:

  • create_response (Boolean) (defaults to: nil) —

    Whether or not to automatically generate a response when a VAD stop event occurs. If interrupt_response is set to false this may fail to create a response if the model is already responding.

    If both create_response and interrupt_response are set to false, the model will never respond automatically but VAD events will still be emitted.

  • idle_timeout_ms (Integer, nil) (defaults to: nil) —

    Optional timeout after which a model response will be triggered automatically. This is useful for situations in which a long pause from the user is unexpected, such as a phone call. The model will effectively prompt the user to continue the conversation based on the current context.

    The timeout value will be applied after the last model response's audio has finished playing, i.e. it's set to the response.done time plus audio playback duration.

    An input_audio_buffer.timeout_triggered event (plus events associated with the Response) will be emitted when the timeout is reached. Idle timeout is currently only supported for server_vad mode.

  • interrupt_response (Boolean) (defaults to: nil) —

    Whether or not to automatically interrupt (cancel) any ongoing response with output to the default conversation (i.e. conversation of auto) when a VAD start event occurs. If true then the response will be cancelled, otherwise it will continue until complete.

    If both create_response and interrupt_response are set to false, the model will never respond automatically but VAD events will still be emitted.

  • prefix_padding_ms (Integer) (defaults to: nil) —

    Used only for server_vad mode. Amount of audio to include before the VAD detected speech (in milliseconds). Defaults to 300ms.

  • silence_duration_ms (Integer) (defaults to: nil) —

    Used only for server_vad mode. Duration of silence to detect speech stop (in milliseconds). Defaults to 500ms. With shorter values the model will respond more quickly, but may jump in on short pauses from the user.

  • threshold (Float) (defaults to: nil) —

    Used only for server_vad mode. Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments.

  • type (Symbol, :server_vad) (defaults to: :server_vad) —

    Type of turn detection, server_vad to turn on simple Server VAD.




# File 'lib/openai/models/realtime/realtime_session.rb', line 640

#self?.variants ⇒ ::Array[OpenAI::Models::Realtime::RealtimeSession::turn_detection]

Returns:

  • (::Array[OpenAI::Models::Realtime::RealtimeSession::turn_detection]) 


443
 
# File 'sig/openai/models/realtime/realtime_session.rbs', line 443

def self?.variants: -> ::Array[OpenAI::Models::Realtime::RealtimeSession::turn_detection]