Benchmarking strategy

[lukaskellerstein]

1. Define configurations for agents

• Choose scenarios that need to be tested
• Translate scenarios as Instructions for Agents
• Define OS state
• Define App state
• Define Evaluation condition = Definition of success

2. Record (Teams) telemetry for scenarios

• create a distilled version of what's happened

3. Define strategy for benchmarking

• How many times each config will run?
• How we will measure / calculate success?

4. Benchmarks

• https://github.com/ltzheng/agent-studio/tree/main
• https://microsoft.github.io/autogen/0.2/blog/2024/01/25/AutoGenBench