voice mode - what is the plan

[radrad]

Hi,

Did you consider using this new feature for your voice mode:
New Voice Mode in Microsoft Foundry | Build Voice Agents Without Code
https://youtu.be/SWYS7vcJ81I?t=263

Are you going to present this project on youtube or a blog post or ebook.

Thanks
Rad

[tommasodotNET]

hi look at #26

[tommasodotNET]

that being said, i don't have plans to present this repo anywhere atm. was thinking about doing a blog post but that's further down the line rn.

[radrad]

Thank you gir merging PR.

I want to build a highly interactie voice agent for learning music theory with piano where I am going to show 2 UI widgets (music sheet on grand staf and virtual midi keyboard) and I can instruct AI agent to show me some music concept let's say:
"Show D# Major scale"
and it would call some tools/mcp servers which should result in notes being displayed in the music sheet widget and as soon as the audio player starts playing notes a Midi keyboard widget will show highlighted notes.
So I would like to have tool calls but to control my AI agent in memory snd I would control the history si it would be very important to have a history as soon as possible.

Do you actually use relay at the backend where you are not expising API key? OR you are using the latest preview feature?
I was wondering if there is a way to have a full control of MAF agent and not to create one in AI Foudry and connect tools in Azure portal but programatically yourself?

What do you mean by:
"this should be changed in the future" below?

'''
will show the transcript live in the same chat as the text one (this should be changed in the future since those messages won't be available to standard chat agent)
'''

What dou you mean by:
"standard chat agent"

What kind of agents exist and what are you using now?

[tommasodotNET]

we have two main agents (that is agents exposed to the frontend, all the others are used by these two as tools) in this repo: the orchestrator and the voice-orchestrator.
the first is a simple chat client agent (that is pure text chat based). the second one is voice based (using realtime models hosted in foundry).
now, what i meant in that comment is that those agents don't share their history (is saved in cosmos using convid, but the voice agent appends "-voice" to the conv id). showing both history in the same ui could confuse the user so it'd be better te separate those.

now when it comes to foundry agent if i get ur q correctly, you don't need to deploy these agents to foundry. foundry is both a resource to host models and a compute resource, where u can define ur own agents setting models, system prompts and a bunch of premade tools.
of course, when u need custom agents, that's not an option. using microsodt agent framework, u could deploy ur own custom agents (full code) to foundry as well, bringing along ur custom tools as well.

that is not what i do in this repo, though. especially with the voice-orchestrator agent that does not rely on maf (if not to invoke agents as tools via a2a maf support). these agents are all built to be hosted in standard .net minimal apis. therefor u can deploy them wherever u want (appservice, k8s, aca, ...). if u want to u can use aspire for deployment, it's rly straightforward.

also both agents handles conv history, saving it to cosmos. if u clone and run using aspire (refer to the readme), we will fire up a local cosmos emulator for u and u can check in the local data explorer that the emulator provides the actual way the conv is saved.