Customize Tracing

Event Tracing for Windows (ETW) can collect data from any of thousands of different 'providers' originating from:

• Certain Applications (Office, Edge, Chrome, ...)
• Windows
• The CPU itself

MSO-Scripts enables specific ETW providers to analyze the behavior and resource usage of Windows Applications.

You can easily customize this in a variety of ways.

ETW Providers

Q. How do I even know which ETW Providers (data sources) are available?
A. ETW has these classes of data providers:

• System Provider
• Event Provider
• Heap Event Provider
• Hardware Counter

Available System Providers are listed here, and their corresponding stackwalk options.

Each of these commands creates a list of Event Providers registered on the current system:

• WPR -providers
• XPerf -providers
  XPerf.exe is often adjacent to WPA.exe, or part of the ADK.
• LogMan query providers
• wevtutil ep and wevtutil gp <Name of Provider>
• PowerShell: Get-NetEventProvider -ShowInstalled | Format-Table -Property Name,GUID

Note

Certain applications make available many additional ETW Event Providers by way of TraceLogging. These are discovered via documentation.

Environment Variables

Extra ETW providers can be enabled (without modifying scripts) by way of two environment variables:

WPT_XPERF = a list of ETW Event Provider definitions in 'XPerf -on' format, separated by '+'

Format: Name|GUID : KeywordFlags : Level : Stack (with additional definitions separated by '+')

• Name|GUID:  Name or GUID of an ETW provider (Mandatory)
• KeywordFlags:  64-bit numeric value indicating which subsets of events to enable (default = all) (Optional)
• Level:  Logging level (0-255): 1=Critical, 2=Error, 3=Warning, 4=Information, 5=Verbose (default) (Optional)
• Stack:  If present, a call stack will be collected for each event. (Optional)

Example: Microsoft-Windows-TCPIP + Microsoft-Windows-RPC:::Stack + e53c6823-7bb8-44bb-90dc-3f86090d48a6:0x00A4:4:Stack
Interpretation: Add 3 ETW providers: TCPIP + RPC with stackwalk + Winsock-AFD with stackwalk, level 4, keyword 0x00A4

WPT_WPRP = a list of WPR Recording Profile descriptions; semicolon-separated
in any of these formats:

• Full path with profile name: c:\MyProfiles\MyProfile.wprp!ProfileName
  To see a list of available Profile Names, run: WPR -Profiles c:\MyProfiles\MyProfile.wprp
• Relative path with profile name: .\WPRP\OtherProfile.wprp!ProfileName (relative to the script's folder)
  or ..\WPRP\OtherProfile.wprp!ProfileName (relative to the root script folder)
• The name of a Built-in Recording Profile, eg. Registry
  To see a list of available Built-in Recording Profiles, run: WPR -Profiles

Tip

Every WPR Profile has a Logging Mode: File and/or Memory. MSO-Scripts defaults to File Logging Mode.
To trace in Memory Logging Mode, run the Trace* Start script with -Loop: TraceCPU Start -Loop ...

Note

Separate WPRP Recording Profiles can create separate recording sessions, which may record duplicate copies of certain events, and could max out WPR's session limit. MSO-Scripts avoids this by giving all <EventCollector> elements the same name: "MSO Event Collector"
See: Controlling the Event Session Name

Modify the Scripts (Advanced)

The MSO-Scripts' PowerShell and XML code is formatted to be relatively easy to modify:

• Add a new WPR Recording Profile (.WPRP file):
  In the Trace*.ps1 script, find the definition of $TraceParams.RecordingProfiles and add a recording profile description:

  • Full path with profile name: c:\MyProfiles\MyProfile.wprp!ProfileName
    To see a list of available Profile Names, run: WPR -Profiles c:\MyProfiles\MyProfile.wprp
  • Relative path with profile name: .\WPRP\OtherProfile.wprp!ProfileName (relative to the script's folder)
    or ..\WPRP\OtherProfile.wprp!ProfileName (relative to the root script folder)
  • The name of a Built-in Recording Profile, eg. Registry
    To see a list of available Built-in Recording Profiles, run: WPR -Profiles

  [See: Authoring Custom WPR Profiles]
  [See the Technical Note above about event duplication when multiple Recording Profiles enable the same providers.]

• Modify an existing WPR Recording Profile (WPRP\*.wprp file):

  • Find and add to the <EventProvider> and <EventProviderId> records.
  • Find and add to the <SystemProvider> and <SystemProviderId> records.

  See: System and Event Provider Definitions and ETW Providers (above)

CPU Monitor Counters (Advanced)

Modern CPUs have many internal event counters, which ETW is able to expose.
These event counters include: Instructions Retired, Cache Misses, Branch Mispredicts, and numerous others.

SHORT STORY

ETW has two mechanisms for capturing CPU event counters:

1. SAMPLING:  Every N CPU events, capture an ETW event with stackwalk.
   Over time, this will generate a statistical picture of where those specific CPU events are happening.
2. GATHERING:  Gather the various CPU counters whenever a specific Windows Kernel event occurs.
   That specific Windows Kernel event is usually a Thread Switch (CSWITCH), so that the CPU counts are associated with the thread which was executing on that logical CPU.

Quick Start:

• To gather-trace Cycles per Instruction Retired (CPI), run one of:

  • MSO-Scripts\BETA\TracePMC Start -CPI
  • MSO-Scripts\BETA\PMC.bat

• To sample-trace Branch Mispredicts and Cache Misses, run one of:

  • MSO-Scripts\BETA\TracePMC Start -PMC *
  • MSO-Scripts\BETA\PMCs.bat

• To view the resulting trace, run:

  • MSO-Scripts\BETA\TracePMC View [-Path <path>\<file>.etl]

  WPA has special views for CPI, Branch Mispredicts, and Cache Misses.

LONG STORY

Windows is pre-configured with a variety of CPU Counters. Others can be added.

• To list the currently available CPU Counters, run one of:
  • WPR -PMCSources
  • XPerf -PMCSources
    If there is only one counter listed then your machine is probably configured for HYPER-V.

Additional CPU-specific Counters can be registered via the Windows Registry.
For Intel CPUs, run:
    MSO-Scripts\BETA\Generate-PmuRegFile -Description
and follow the instructions at the end.

• To sample-trace specific CPU counters, run one of:
  • MSO-Scripts\BETA\TracePMC Start -PMC Counter1[,Counter2[,...]]
  • MSO-Scripts\BETA\PMCs.bat Counter1 [Counter2 [...]]
• To gather-trace specific CPU counters at each Thread Switch (CSWITCH) event, run:
  • MSO-Scripts\BETA\PMC.bat Counter1 [Counter2 [...]]
• To view the resulting trace, run one of:
  • MSO-Scripts\BETA\TracePMC View [-Path <path>\<file>.etl]
  • WPA <path>\<file>.etl -profile MSO-Scripts\BETA\WPAP\CPUCounters.wpaProfile

FURTHER READING

• Recording PMU Events
• Recording Hardware Performance (PMU) Events with Complete Examples
• Intel CPU Metrics Reference
• TraceEvent – Monitoring and Diagnostics for the Cloud