Support OpenAI's Realtime API

[codearranger]

It looks like this module is very close to working with openai's new realtime API

https://platform.openai.com/docs/guides/realtime

[xiaozhao1795]

I'm also researching how to implement OpenAI's new realtime API. What are some ways to achieve this, considering that this module does not support streaming speech playback?

[amigniter]

Even with speech playback added (to the same channel audio is streamed from) still some sort of a middle man will be required to pack JSON to and from the OpenAI and back to FS.
Too bad they do not offer some sort of developer's preview, I can't even test the damn thing without paying it. And still I have no idea how much it will cost me to develop a connector, and lots of tests need to be done.

[xiaozhao1795]

Even with speech playback added (to the same channel audio is streamed from) still some sort of a middle man will be required to pack JSON to and from the OpenAI and back to FS. Too bad they do not offer some sort of developer's preview, I can't even test the damn thing without paying it. And still I have no idea how much it will cost me to develop a connector, and lots of tests need to be done.

Hello, I can help you with testing. I have written a simple WebSocket relay application. The mod_audio_stream connects to the WebSocket, sends data to the relay application, and the relay application forwards the data to OpenAI. OpenAI then returns the audio stream to FreeSWITCH. Subsequently, mod_audio_stream can directly connect to OpenAI in real-time.

async def receive_from_source_and_send_to_target(websocket, target_ws):
async for message in target_ws:
message_dict = json.loads(message)
type_ = message_dict.get("type")
if "response.audio.delta" == type_:
audio = {
"type": "streamAudio",
"data": {
"audioDataType": "raw",
"sampleRate": 8000,
"audioData": message_dict.get("delta")
}
}
await websocket.send(json.dumps(audio))
elif "response.content_part.done" == type_:
print("Received------->" + message)

Here, target_ws refers to OpenAI real-time, and websocket refers to the server connected by mod_audio_stream.

[amigniter]

As discussed in issue 56, this feature is closely related to the idea of integrating real-time speech-to-speech capabilities. While it wasn't part of my original plan for this module, the numerous requests from users—many of whom rely on this for both personal and commercial use—have led me to reconsider. Thanks to @eschmidbauer for inspiring this decision.

I want to clarify that this module has always been a personal project, made available to the community as a contribution to open-source. I have not gained financial benefit from it, nor do I expect any. My goal has always been to help improve the ecosystem and provide value to those who use it.

That said, integrating real-time playback of audio from memory (as part of a speech-to-speech workflow) requires substantial ongoing work. Given the commercial nature of many speech-to-speech services, including OpenAI's ChatGPT real-time API, I am seeking to become a sponsored project on GitHub. If you're willing to support this development, you can sponsor my project on GitHub. Sponsors will receive access to a private repository where they'll find exclusive features like automatic playback, early access to new functionality, and more.

I truly appreciate your understanding and support, and I look forward to continuing to contribute to the community with this exciting new feature!

[amigniter]

@xiaozhao1795 if you are willing to contribute and test please contact me and i will give you access (once it's ready and reaches testing phase) so we can proceed with some tests.

[avimar]

new version mod-audio-stream v1.0.3 - 🚀 Introducing Bi-Directional Streaming with automatic playback

Does this work directly with OpenAI's realtime endpoint? How would I configure my prompt and be able to handle any other messages from the endpoint?

Sorry for bugging you, this seems to be the closest information online on how to integrate freeswitch with OpenAI's realtime.

[amigniter]

@avimar

Does this work directly with OpenAI's realtime endpoint? How would I configure my prompt and be able to handle any >other messages from the endpoint?
Sorry for bugging you, this seems to be the closest information online on how to integrate freeswitch with OpenAI's realtime.

Directly, no. This module is designed to be engine-agnostic and it does not integrate directly with the engines such as OpenAI real-time API or ElevenLabs Conversational AI, so any engine can be used with it. For that reason you put a WebSocket server in front of it, a proxy, and whatever the speech-to-speech provider returns, you'd repack and send back to the module.
Take a look at the README and README.playback, everything you need you would find there.

[andrewwmangold]

@xiaozhao1795 if you are willing to contribute and test please contact me and i will give you access (once it's ready and reaches testing phase) so we can proceed with some tests.

I'm willing to up my sponsorship and contribute to this project. I am eager to test dev builds and contribute to the effort to streamline this module, as it has much promise, as I see it.

[NormandoHall]

A dummy question. It is need it to use mod_audio_stream when Open IA has SIP Realtime?