Add W365 Computer Use sample agent#305
Open
desmarest wants to merge 1 commit into
Open
Conversation
Dependency ReviewThe following issues were found:
License Issuesdotnet/w365-computer-use/sample-agent/W365ComputerUseSample.csproj
OpenSSF ScorecardScorecard details
Scanned Files
|
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| return; | ||
| } | ||
|
|
||
| string[] parts = authorizationHeader?.Split(' ')!; |
| { | ||
| context.Options.TokenValidationParameters.ConfigurationManager = _openIdMetadataCache.GetOrAdd(validationOptions.AzureBotServiceOpenIdMetadataUrl, key => | ||
| { | ||
| return new ConfigurationManager<OpenIdConnectConfiguration>(validationOptions.AzureBotServiceOpenIdMetadataUrl, new OpenIdConnectConfigurationRetriever(), new HttpClient()) |
| { | ||
| context.Options.TokenValidationParameters.ConfigurationManager = _openIdMetadataCache.GetOrAdd(validationOptions.OpenIdMetadataUrl, key => | ||
| { | ||
| return new ConfigurationManager<OpenIdConnectConfiguration>(validationOptions.OpenIdMetadataUrl, new OpenIdConnectConfigurationRetriever(), new HttpClient()) |
| private static async Task<(string agentId, string tenantId)> ResolveTenantAndAgentId(ITurnContext turnContext, UserAuthorization authSystem, string authHandlerName) | ||
| { | ||
| string agentId = ""; | ||
| if (turnContext.Activity.IsAgenticRequest()) |
Comment on lines
+368
to
+375
| if (type == "function_call" || type == "function_call_output") | ||
| { | ||
| if (item.TryGetProperty("call_id", out var idProp) | ||
| && cuaOnlyCallIds.Contains(idProp.GetString() ?? string.Empty)) | ||
| { | ||
| return false; | ||
| } | ||
| } |
Comment on lines
+214
to
+218
| catch (Exception ex) | ||
| { | ||
| _logger.LogWarning(ex, "CUA intent classifier threw — defaulting to needsCua=true."); | ||
| return true; | ||
| } |
Comment on lines
+506
to
+509
| catch (Exception ex) | ||
| { | ||
| logger.LogWarning(ex, "Failed to end W365 session"); | ||
| } |
Comment on lines
+47
to
+50
| catch (Exception ex) | ||
| { | ||
| logger?.LogWarning("There was an error registering for observability: {Message}", ex.Message); | ||
| } |
Comment on lines
+865
to
+868
| catch (Exception ex) | ||
| { | ||
| logger.LogWarning(ex, "Best-effort EndSession during recovery failed"); | ||
| } |
Comment on lines
+931
to
+935
| catch (Exception ex) | ||
| { | ||
| _logger.LogError(ex, "Function call {Name} threw. call_id={CallId}", name, callId); | ||
| return CreateFunctionOutput(callId, $"Error: {ex.Message}"); | ||
| } |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a new .NET sample agent (dotnet/w365-computer-use/) that demonstrates running a “computer use” loop against a Windows 365 Cloud PC via the W365 Computer Use MCP server, including optional function-tool routing and OneDrive screenshot upload/sharing.
Changes:
- Introduces a new Agent Framework sample project/solution with configuration, local/production setup docs, and deployment guidance.
- Implements a CUA orchestrator that translates
computer_callactions into MCP tool invocations, captures screenshots, and iterates until completion/end-session. - Adds basic OpenTelemetry wiring + custom Activity/Meter helpers and an ASP.NET JWT token validation helper.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| dotnet/w365-computer-use/W365ComputerUseSample.sln | New solution file wiring up the sample-agent project. |
| dotnet/w365-computer-use/sample-agent/W365ComputerUseSample.csproj | New web project with Agent Framework + MCP + OTEL dependencies. |
| dotnet/w365-computer-use/sample-agent/ToolingManifest.json | Declares the W365 MCP server metadata (reference material for tooling). |
| dotnet/w365-computer-use/sample-agent/telemetry/AgentMetrics.cs | Adds custom Activity/Meter helpers for agent/http operations (contains a critical async/finalization bug). |
| dotnet/w365-computer-use/sample-agent/telemetry/A365OtelWrapper.cs | Wraps observed operations with baggage + observability token cache registration (has agentId fallback bug). |
| dotnet/w365-computer-use/sample-agent/ServiceExtensions.cs | Adds OpenTelemetry resource/tracing/metrics configuration for the sample. |
| dotnet/w365-computer-use/sample-agent/README.md | End-to-end documentation (setup, Playground testing, production deployment, troubleshooting). |
| dotnet/w365-computer-use/sample-agent/Properties/launchSettings.json | Local dev profile for http://localhost:3978. |
| dotnet/w365-computer-use/sample-agent/Program.cs | ASP.NET host setup, /api/messages + /api/health endpoints, and agent registration. |
| dotnet/w365-computer-use/sample-agent/nuget.config | Pins package source configuration to nuget.org. |
| dotnet/w365-computer-use/sample-agent/ComputerUse/Models/ComputerUseModels.cs | DTOs for Azure OpenAI Responses “computer use” request/response/tool definitions. |
| dotnet/w365-computer-use/sample-agent/ComputerUse/ICuaModelProvider.cs | Abstraction for sending serialized Responses API requests. |
| dotnet/w365-computer-use/sample-agent/ComputerUse/ComputerUseOrchestrator.cs | Core CUA loop, action→MCP mapping, screenshot capture, OneDrive upload/share (has a couple concrete runtime/URL/cancellation issues). |
| dotnet/w365-computer-use/sample-agent/ComputerUse/AzureOpenAIModelProvider.cs | Azure OpenAI Responses API client using API key auth. |
| dotnet/w365-computer-use/sample-agent/AspNetExtensions.cs | Adds JWT bearer validation helper (contains token parsing robustness issues). |
| dotnet/w365-computer-use/sample-agent/appsettings.json | Default configuration with placeholders for AOAI + connection + screenshot settings. |
| dotnet/w365-computer-use/sample-agent/Agent/MyAgent.cs | Agent turn handling: intent classification fast-path, tool loading, CUA orchestration, streaming updates (has async-callback misuse). |
| dotnet/w365-computer-use/sample-agent/a365.config.example.json | Example a365 CLI config template with placeholders. |
| dotnet/w365-computer-use/sample-agent/.gitignore | Ignores local overrides, screenshots, and deploy artifacts for this sample. |
Comment on lines
+86
to
+110
| public static Task InvokeObservedAgentOperation(string operationName, ITurnContext context, Func<Task> func) | ||
| { | ||
| MessageProcessedCounter.Add(1); | ||
| var activity = InitializeMessageHandlingActivity(operationName, context); | ||
| var stopwatch = Stopwatch.StartNew(); | ||
| try | ||
| { | ||
| return func(); | ||
| } | ||
| catch (Exception ex) | ||
| { | ||
| activity?.SetStatus(ActivityStatusCode.Error, ex.Message); | ||
| activity?.AddEvent(new ActivityEvent("exception", DateTimeOffset.UtcNow, new() | ||
| { | ||
| ["exception.type"] = ex.GetType().FullName, | ||
| ["exception.message"] = ex.Message, | ||
| ["exception.stacktrace"] = ex.StackTrace | ||
| })); | ||
| throw; | ||
| } | ||
| finally | ||
| { | ||
| stopwatch.Stop(); | ||
| FinalizeMessageHandlingActivity(activity, context, stopwatch.ElapsedMilliseconds, true); | ||
| } |
Comment on lines
+119
to
+128
| if (parts.Length != 2 || parts[0] != "Bearer") | ||
| { | ||
| context.Options.TokenValidationParameters.ConfigurationManager ??= options.ConfigurationManager as BaseConfigurationManager; | ||
| await Task.CompletedTask.ConfigureAwait(false); | ||
| return; | ||
| } | ||
|
|
||
| JwtSecurityToken token = new(parts[1]); | ||
| string issuer = token.Claims.FirstOrDefault(claim => claim.Type == AuthenticationConstants.IssuerClaim)?.Value!; | ||
|
|
Comment on lines
+71
to
+73
| agentId = agentId ?? Guid.Empty.ToString(); | ||
| string? tempTenantId = turnContext?.Activity?.Conversation?.TenantId ?? turnContext?.Activity?.Recipient?.TenantId; | ||
| string tenantId = tempTenantId ?? Guid.Empty.ToString(); |
| if (_toolType == "computer" && session.ConversationHistory.Count == 0 && session.SessionStarted) | ||
| { | ||
| var initialScreenshot = await CaptureScreenshotAsync(w365Tools, session.W365SessionId, cancellationToken); | ||
| var initialName = $"{conversationId[..8]}_{++session.ScreenshotCounter:D3}_initial"; |
Comment on lines
+1010
to
+1012
| var driveBase = string.IsNullOrEmpty(_oneDriveUserId) | ||
| ? "https://graph.microsoft.com/v1.0/me/drive" | ||
| : $"https://graph.microsoft.com/v1.0/users/{_oneDriveUserId}/drive"; |
Comment on lines
+1018
to
+1024
| using var request = new HttpRequestMessage(HttpMethod.Put, url); | ||
| request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", graphAccessToken); | ||
| request.Content = new ByteArrayContent(Convert.FromBase64String(base64Data)); | ||
| request.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("image/png"); | ||
|
|
||
| var response = await _httpClient.SendAsync(request); | ||
| if (response.IsSuccessStatusCode) |
Comment on lines
+162
to
+173
| var (_, nonCuaAdditionalTools) = await GetToolsAsync(turnContext, ToolAuthHandlerName); | ||
| var directResponse = await _orchestrator.RunAsync( | ||
| conversationId, | ||
| userText, | ||
| w365Tools: [], | ||
| additionalTools: nonCuaAdditionalTools, | ||
| graphAccessToken: null, | ||
| onStatusUpdate: status => turnContext.StreamingResponse.QueueInformativeUpdateAsync(status).ConfigureAwait(false), | ||
| onCuaStarting: null, | ||
| onFolderLinkReady: null, | ||
| includeCuaTool: false, | ||
| cancellationToken: cancellationToken); |
Comment on lines
+209
to
+216
| var response = await _orchestrator.RunAsync( | ||
| conversationId, | ||
| userText, | ||
| w365Tools, | ||
| additionalTools: additionalTools, | ||
| graphAccessToken: graphToken, | ||
| onStatusUpdate: status => turnContext.StreamingResponse.QueueInformativeUpdateAsync(status).ConfigureAwait(false), | ||
| onCuaStarting: async (isNewSession) => |
Adds dotnet/w365-computer-use/ as a new sample alongside agent-framework,
autonomous, and semantic-kernel. The sample demonstrates an Agent 365
agent that controls a Windows 365 Cloud PC via the W365 Computer Use MCP
server using Azure OpenAI's computer-use model.
Highlights:
- Intent classifier skips the W365 session-acquisition handshake on
non-CUA messages (chit-chat, mail/calendar function-tool requests).
- Full CUA loop translates model computer_call actions to MCP tool
invocations, captures screenshots, and feeds them back to the model
until it emits OnTaskComplete or EndSession.
- Per-prompt OneDrive screenshot folders ({yyyy-MM-dd}/{HHmmss}_{slug})
with org-scoped sharing links surfaced to the user.
- Filters ATG's synthetic "Error" sentinel tool out of the LLM tool list
to avoid Azure OpenAI invalid_function_parameters rejections, while
still reading its description for user-facing error messaging.
- Supports both computer-use-preview and gpt-5.4 / gpt-5.4-mini.
desmarest
force-pushed
the
users/bertd/w365-computer-use-sample
branch
from
ffe7c23 to
8725549
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
dotnet/w365-computer-use/as a new sample alongsideagent-framework,autonomous, andsemantic-kernel.The sample demonstrates an Agent 365 agent that controls a Windows 365 Cloud PC via the W365 Computer Use MCP server using Azure OpenAI's computer-use model (
computer-use-previewandgpt-5.4/gpt-5.4-miniare both supported).Highlights
computer_callactions into MCP tool invocations, captures screenshots, and feeds them back to the model until it emitsOnTaskCompleteorEndSession.{yyyy-MM-dd}/{HHmmss}_{slug}) with org-scoped sharing links surfaced to the user inline.Errorsentinel tool out of the LLM tool list to avoid Azure OpenAIinvalid_function_parametersrejections, while still reading its description for user-facing error messaging.appsettings.Development.jsonClientSecret override) and full Azure App Service production deployment (resource creation, App Service config withAuthType=ClientSecret+AuthorityEndpoint,dotnet publish+az webapp deploy, Bot Channels Registration in Foundry, blueprint linking, anda365 publishfor Teams app upload).a365.config.example.jsoncommitted as a placeholder template since the CLI no longer ships aconfig initcommand.