OPENAI GPT - REALTIME CONVERSATION (FRONT-END DESKTOP & NATIVE MOBILE)
============================================
OPENAI GPT - REALTIME CONVERSATION (FRONT-END DESKTOP & NATIVE MOBILE) - ELEMENT DESCRIPTION
------------------------------------------------------------------------------
OPENAI GPT - REALTIME CONVERSATION (FRONT-END DESKTOP & NATIVE MOBILE) provides GPT realtime multimodal conversation capabilities, including audio, camera, screen sharing inputs.
STEP-BY-STEP SETUP
--------------------------------
0) Register on OpenAI and get your OPENAI API KEY. If your users are behind strict firewalls or restrictive networks, optionally register on METERED.CA to obtain a TURN SERVER SECRET KEY.
1) Test on
https://platform.openai.com/playground/realtime to confirm your account and key may use REALTIME API
2) Implement the OPENAI GPT - REALTIME CONVERSATION (FRONT-END DESKTOP & NATIVE MOBILE) ERROR workflow (see demo) in order to raise any OpenAI errors in your application.
3) Register on plugins.wiseable.io. Create a new Credential which associates your BUBBLE APP URL your OPENAI API KEY, and optionally yoour METERED.CA TURN SERVER SECRET KEY.
The registration service will generate your PUBLIC ACCESS KEY. This key serves as a secure proxy for your real API key. It allows your application to communicate with the service without exposing your real API key. Since this PUBLIC ACCESS KEY is explicitly tied to your registered BUBBLE APP URL, it can only be used from that domain, ensuring that even if the key is publicly visible, it remains safe and cannot be misused by unauthorized sources.
4) In the Plugin Settings, enter your PUBLIC ACCESS KEY generated at the previous step.
5) Add the OPENAI GPT - REALTIME CONVERSATION (FRONT-END DESKTOP & NATIVE MOBILE) ELEMENT to the page on which the chat must be integrated. Select the RESULT DATA TYPE as CONVERSATION (OPENAI GPT REALTIME).
6) Integrate the logic into your application using the following OPENAI GPT REALTIME CONVERSATION element's states and actions:
FIELDS:
- RESULT DATA TYPE: Must always be selected as CONVERSATION (OPENAI GPT REALTIME).
- INPUT DEVICES TYPE : Allow audio & video devices type to be used, or only audio ones. Valid values: AUDIO | AUDIO & VIDEO
- MICROPHONE : Name of the Microphone to use as a source. It must be one of the item of INPUT MICROPHONES state.
- CAMERA : Name of the Camera to use as a source. It must be one of the item of INPUT CAMERAS state.
REUSABLE PROMPT
- PROMPT ID : Unique identifier of your prompt, found in your OpenAI dashboard.
- PROMPT VERSION : A specific version of your prompt (defaults to the "current" version as specified in your OpenAI dashboard).
PROMPT (MANUAL)
- SYSTEM PROMPT : Define the role of the AI assistant. You may hide thus data from eavesdroppers by using GET DATA FROM AN EXTERNAL API > OPENAI CHATGPT - ENCRYPT and set this field as result of this API.
- TRANSCRIPTION MODEL: Model to use for input transcription. Valid models are: gpt-4o-transcribe | gpt-4o-mini-transcribe | whisper-1. Defaults to whisper-1
AUDIO SETTINGS :
- ECHO CANCELLATION : Echo cancellation is a feature which attempts to prevent echo effects on a two-way audio connection by attempting to reduce or eliminate crosstalk between the user's output device and their input device.
- NOISE SUPPRESSION : Noise suppression automatically filters the audio to remove background noise, hum caused by equipment, and the like from the sound before delivering it to your code.
- AUTO GAIN CONTROL : Automatic gain control is a feature in which a sound source automatically manages changes in the volume of its source media to maintain a steady overall volume level.
- LANGUAGE : The language of the input audio. Supplying the input language in ISO-639-1 (e.g. en) format will improve accuracy and latency. Leave empty for automated detection.
APP DATA RETRIEVAL SETTINGS (FRONT-END DESKTOP ONLY) :
- APP DATA AGENT : Set to yes to activate data retrieval from your app when needed to answer to the user prompt. Your app DATA API must be enabled.
- DATA AGENT MODEL : Model used by App Data Agent.
METERED.CA NETWORK SETTINGS :
- TURN SERVER ROUTING : If true, the connection will be routed through a Metered.ca TURN server. Your Public Access Key must be configured with a valid Metered.ca Secret Key.
- CUSTOM DOMAIN : Metered.ca custom domain associated with your Metered.ca Secret Key. Mandatory if TURN Server Routing is true. Example: customdomain.metered.live
EVENTS:
- ERROR: Event triggered when an error occurs.
- CALL TOOL: Event triggered when a function is triggered, as defined in TOOLS of START CONVERSATION action.
- CONVERSATION TO SAVE: Event triggered when any of the conversation has changed.
- CONVERSATION READY TO START: Event triggered when a conversation is ready to start.
- TOKEN USAGE UPDATE : Event triggered when a token usage update is reported.
- RATE LIMITS UPDATE : Event triggered when a token limit update is reported.
- AUDIO RECORDING SAVED (FRONT-END DESKTOP ONLY) : Event triggered when an audio recording has been successfully saved. Populates LATEST AUDIO RECORDING URL states.
- ELEMENT SCREENSHOT SENT (FRONT-END DESKTOP ONLY) : Event triggered when a screenshot of the element has been sent.
EXPOSED STATES:
Use any element able to show/process the data of interest (such as a Group with a Text field) stored within the result of the following states of the OPENAI GPT - REALTIME CONVERSATION (FRONT-END DESKTOP & NATIVE MOBILE) ELEMENT:
- ERROR: Error message upon Error event trigger.
- IS LISTENING: Returns true when listening is in progress.
- IS AI SPEAKING: Returns true when AI speaking is in progress.
- IS RECORDING (FRONT-END DESKTOP ONLY): Returns true when audio recording is in progress.
- LATEST AUDIO RECORDING URL (FRONT-END DESKTOP ONLY): Returns the latest audio recording URL. Populates upon AUDIO RECORDING SAVED event.
- IS MICROPHONE MUTED: Returns true when microphone is muted.
- IS AI WATCHING CAMERA: Returns true when camera watching is in progress.
- IS AI WATCHING SCREEN (FRONT-END DESKTOP ONLY): Returns true when screen watching is in progress.
- IS CAMERA ON: Returns true when camera is on.
- IS SCREEN SHARING ON (FRONT-END DESKTOP ONLY): Returns true when screen sharing is on.
- TOOL NAME: Name of the tool, set upon CALL TOOL event.
- TOOL ARGUMENTS: Arguments of the tool, set upon CALL TOOL event.
- TOOL CALL ID: Unique identifier of the tool call, set upon CALL TOOL event.
- INPUT MICROPHONES: List of detected microphones, populated after DETECT DEVICES action.
- INPUT CAMERAS : List of detected cameras, populated after DETECT DEVICES action.
- ALL CONVERSATIONS: List of conversations containing a list of role and message content.
- ALL CONVERSATIONs (RAW DATA): String containing All Conversations in JSON format. You may use this string to load threads in "LOAD CONVERSATIONS" action.
- LATEST INPUT TOKEN USAGE: Latest input token usage of the AI engine. Triggered upon TOKEN USAGE UPDATE event.
- LATEST OUTPUT TOKEN USAGE: Latest output token usage of the AI engine. Triggered upon TOKEN USAGE UPDATE event.
- LATEST REQUESTS RATE LIMIT REMAINING COUNT: Latest remaining requests count rate limit. Triggered upon RATE LIMIT UPDATE event.
- LATEST REQUESTS RATE LIMIT RESET SECONDS: Latest seconds count until the tokens rate limit resets. Triggered upon RATE LIMIT UPDATE event.
- LATEST TOKENS RATE LIMIT REMAINING COUNT: Latest remaining tokens count rate limit. Triggered upon RATE LIMIT UPDATE event.
- LATEST TOKENS RATE LIMIT RESET SECONDS: Latest seconds count until the tokens rate limit resets. Triggered upon RATE LIMIT UPDATE event.
ELEMENT ACTIONS - TRIGGERED IN WORKFLOW:
- DETECT DEVICES: Detect input devices based on INPUT DEVICES TYPE input field. Populate relevant INPUT CAMERAS and INPUT MICROPHONES states.
- MUTE CURRENT MICROPHONE: Mute the current microphone.
- UNMUTE CURRENT MICROPHONE: Unmute the current microphone.
- START CONVERSATION: Start Voice Conversation.
Inputs Fields:
- VOICE: The voice to use when generating the audio modality. Valid values: alloy | echo | shimmer | ash | ballad | coral | sage | verse.
- MODALITIES: The set of modalities the model can respond with.Valid values : text | text,audio
- RESUMED CONVERSATION ID: Must be one of the CONVERSATION ID of ALL CONVERSATIONS state.
- TEMPERATURE: Sampling temperature to use, between 0 and 2. Higher values mean the model will take more risks. Try 0.9 for more creative applications, and 0 for ones with a well-defined answer.
- TOOLS: Array containing function definitions. See
https://cookbook.openai.com/examples/how_to_call_functions_with_chat_models#basic-concepts.
- TEMPERATURE: What sampling temperature to use, between 0 and 2. Higher values mean the model will take more risks. Try 0.9 for more creative applications, and 0 for ones with a well-defined answer.
- SEND TEXT MESSAGE : Send a message in text format.
Inputs Fields:
- MESSAGE : The message in text format.
- STOP CONVERSATION: Stop Voice Conversation.
Inputs Fields:
- INTERRUPT PLAYBACK : If true, immediately interrupt playback. If false, buffered audio will continue playing until completed..
- SET CONVERSATION TITLE : Set a custom Conversation Title of a given Conversation ID.
Inputs Fields :
- CONVERSATION ID : Set the title to this Conversation ID. A valid Conversation ID is one of the exposed state ALL CONVERSATIONS. - DELETE CONVERSATION ID : Delete the specified CONVERSATION ID.
Inputs Fields :
- CONVERSATION ID : Delete the specified CONVERSATION ID.
- LOAD CONVERSATIONS : Load the conversations.
Inputs Fields :
- CONVERSATIONS (RAW DATA) : String formatted as JSON-safe, containing the conversations in JSON format.
- DELETE CONVERSATION ID : Delete the specified CONVERSATION ID.
Inputs Fields :
- CONVERSATION ID : Delete the specified CONVERSATION ID.
- UPDATE SESSION SETTINGS : Dynamically update conversation settings during the session.
Inputs Fields:
- TOOLS: Array containing functions definition. See
https://cookbook.openai.com/examples/how_to_call_functions_with_chat_models#basic-concepts - TOOLS CHOICE: How the model chooses tools. Options are "auto", "none", "required", or specify a function.
- TEMPERATURE: What sampling temperature to use, between 0 and 2. Higher values mean the model will take more risks. Try 0.9 for more creative applications, and 0 for ones with a well-defined answer.
- SEND TOOL RESULT: Send the tool results. See
https://cookbook.openai.com/examples/how_to_call_functions_with_chat_models#basic-concepts.
Inputs Fields:
- TOOL CALL ID: Unique identifier of the tool call from the TOOL CALL ID state, which is set upon CALL TOOL event.
- TOOL RESULT: Result of the tool to pass.
- START RECORDING (FRONT-END DESKTOP ONLY): Start to record audio
- STOP RECORDING (FRONT-END DESKTOP ONLY): Stop to record audio
- SAVE RECORDING (FRONT-END DESKTOP ONLY): Save the latest audio recording to your app. The audio format will be automatically selected based on browser's capabilities.
Inputs Fields:
- FILENAME: Filename without extension. The extension will automatically be set based on the file type
- ATTACH TO RESULT: Optional thing to privately attach the file to.
- MUTE MICROPHONE: Start to record audio
- UNMUTE MICROPHONE: Stop to record audio
- SEND SCREENSHOT OF ELEMENT: Send the screenshot of an element on the same page with the specified Element ID
- ELEMENT ID: The ID of the HTML element to capture as a screenshot.
- START SCREEN SHARING (FRONT-END DESKTOP ONLY): Start Screen Sharing
- STOP SCREEN SHARING (FRONT-END DESKTOP ONLY): Stop Screen Sharing
- START CAMERA: Start Camera
- STOP CAMERA: Stop Camera
IMPLEMENTATION EXAMPLE
======================
Feel free to browse the app editor in the Service URL for an implementation example.
TROUBLESHOOTING
================
Any plugin-related error will be posted to the Logs tab, "Server logs" section of your App Editor.
Make sure that "Plugin server side output" and "Plugin client side output" are selected in "Show Advanced".
> Server Logs Details:
https://manual.bubble.io/core-resources/bubbles-interface/logs-tab#server-logsPERFORMANCE CONSIDERATIONS
===========================
N/A
QUESTIONS?
===========
Contact us at
[email protected] for any additional features you would require or support questions.
Built on Bubble
