Build Quality Checks gives unexpected baseline coverage percentage

[QxBytes]

Describe the context

• Extension: BuildQualityChecks
• Environment: Azure DevOps Services
  • Server version: N/A
• Agent type: Microsoft
  • Agent version: N/A
• Pipeline type: YAML

Describe the problem and expected behavior
When I run BQC twice for the same code I expect the baseline code coverage % to be what BQC reports as the current code coverage in the previous run; however this is not the case and I don't know how BQC gets the previous run's code coverage % (what files it looks for etc.)

Steps:

• I run my golang unit test suite and generate a coverage profile (coverage.out)
• I use go tool cover -func=coverage.out to confirm the % covered
• I convert the .out to .json with gocov convert
• I convert the .json to .xml with gocov-xml
• I publish the xml file and upon downloading, I confirm the % covered
• I publish the xml file using PublishCodeCoverageResults@2
• I run BuildQualityChecks@8 and it shows the same % covered (Shows # of lines and # of lines covered)
• I wait for the pipeline to finish and the code coverage tab also reports the same % covered
• Once the pipeline finishes and I have a baseline, I run it again without making any changes to the code; however, when it reaches the check, the # of lines and # of lines covered is far less and the % covered differs greatly.
• I expect that if I run BuildQualityChecks and get a % covered w/ # of lines and # of lines covered, the next time I run it, the baseline shows the previous run's % covered w/ # of lines and # of lines covered.
• I confirm the baseline build ID corresponds to the expected previous run

BQC Logs from second run (The first run shows the current coverage still as 56.11% with same line counts):

Starting: Check Code Coverage Regression
==============================================================================
Task         : Build Quality Checks
Description  : Breaks a build based on quality metrics like number of warnings or code coverage.
Version      : 8.2.1
Author       : Microsoft
Help         : [[Docs]](https://github.com/MicrosoftPremier/VstsExtensions/blob/master/BuildQualityChecks/en-US/overview.md)
...
Validating code coverage policy...
Successfully read code coverage data from build.
Evaluating coverage data from 1 filtered code coverage data sets...
Total lines: 1702
Covered lines: 955
Code Coverage (%): 56.1105
Found baseline build with ID 120470974.
Successfully read code coverage data from build.
Evaluating coverage data from 1 filtered code coverage data sets...
Total lines: 687
Covered lines: 269
Code Coverage (%): 39.1557
[SUCCESS] Code coverage policy passed with 56.1105% (955/1702 lines).

Not sure how we get 39.15%-- if BQC is just reading the previously published xml data shouldn't it still be 56.11%?

Artifacts folder structure:

XML Uploaded:

Code Coverage Tab:

YAML:

- stage: code_coverage
    displayName: Code Coverage Check
    dependsOn:
      - test
    jobs:
      - job: coverage
        displayName: Check Coverage
        pool:
          name: "$(BUILD_POOL_NAME_DEFAULT)"
        variables:
          system.debug: true
          BQC.LogRawData: true
        steps:
          - task: DownloadPipelineArtifact@2
            inputs:
              artifact: 'linux-coverage'
              path: './'
          - bash: |
              ... make tools ...

              gocov convert linux-coverage.out > linux-coverage.json
              gocov-xml < linux-coverage.json > linux-coverage.xml

              mkdir coverage

              mv linux-coverage.xml coverage/
            name: "Coverage"
            displayName: "Generate Coverage Report"
            condition: always()
          - task: UseDotNet@2
            displayName: 'Use .NET Core sdk 7.0.x for Publish Code Coverage Results v2'
            inputs:
              version: 7.0.x
          - task: PublishCodeCoverageResults@2
            displayName: "Publish Code Coverage Report"
            condition: always()
            inputs:
              summaryFileLocation: coverage/linux-coverage.xml
          - task: PublishPipelineArtifact@1
            inputs:
              targetPath: 'coverage/linux-coverage.xml'
              artifactName: 'linux-coverage-xml'
          - task: BuildQualityChecks@8
            displayName: "Check Code Coverage Regression"
            condition: always()
            inputs:
              checkCoverage: true
              coverageFailOption: "build"
              coverageType: "lines"
              fallbackOnPRTargetBranch: false
              baseBranchRef: "my-feature-branch"
              allowCoverageVariance: true
              coverageVariance: 0.25

Task logs
Run your pipeline with the following variables:

• For BuildQualityChecks: System.Debug and BQC.LogRawData set to true
• For CreateWorkItem: System.Debug set to true
• For PostBuildCleanup: System.Debug and PBC.LogRawData set to true

Send the task log to PSGerExtSupport@microsoft.com and reference your GitHub issue.

Attention: The log file may contain sensitive data (e.g., server or organization names, project names, variable information). Please do not attach the log to your GitHub issue and or remove the information from the log file before attaching or sending.

[ReneSchumacher]

Hi @QxBytes,

BQC is not reading the XML file itself. It uses the Azure DevOps code coverage summary API to get the coverage values. So, you should not look at the numbers in the Code Coverage tab (unfortunately) as they may differ. The Code Coverage tab is generated using the ReportGenerator component/library, which has its own logic for parsing coverage files.

Please compare coverage values from BQC only to the values shown on the build summary page or the API. The summary page always shows line coverage:

The API (https://dev.azure.com/{organization}/{project}/_apis/test/codecoverage?buildId={buildId}) shows all the data:

{
  "coverageData": [
    {
      "coverageStats": [
        {
          "label": "Branches",
          "position": 6,
          "total": 509,
          "covered": 458,
          "isDeltaAvailable": false,
          "delta": 0
        },
        {
          "label": "Lines",
          "position": 4,
          "total": 1083,
          "covered": 1026,
          "isDeltaAvailable": false,
          "delta": 0
        }
      ],
      "buildPlatform": "",
      "buildFlavor": ""
    }
  ],
  "build": {
    "id": "7019",
    "url": "https://dev.azure.com/{organization}/_apis/build/Builds/{buildId}"
  },
  "deltaBuild": null,
  "status": "finalized",
  "coverageDetailedSummaryStatus": "noModulesFound"
}

Since you have the baseline build ID in the logs, could you please query the API for that build ID? The results should match what you see in BQC. You can also open the build summary for that particular baseline build by opening https://dev.azure.com/{organization}/{project}/_build/results?buildId={buildId}&view=results in the browser. You should find the same values on the summary page and similar values in the Code Coverage tab.

I can see that you have specified a baseBranchRef in your BQC configuration. In this case, BQC always compares with the latest build for that particular branch, which is not necessarily the build you just executed. To illustrate that, let's assume that you have a single build definition that runs for different branches:

Existing branches:

• main (refs/heads/main) - build was executed
• feat1 (refs/heads/feat1) - build was executed
• feat2 (refs/heads/feat2) - build was not executed

Coverage values were as shown below:

Branch	Coverage	ID
refs/heads/main	50%	1
refs/heads/feat1	60%	2

When executing the pipeline, the compare value depends on which branch the pipeline runs for and how baseBranchRef and fallbackOnPRTargetBranch are set (samples below):

Branch of the build run	baseBranchRef	fallbackOnPRTargetBranch	Baseline build ID	Baseline coverage
refs/heads/main	not set	ignored	1	50%
refs/heads/main	refs/heads/main	ignored	1	50%
refs/heads/main	refs/heads/feat1	ignored	2	60%
refs/heads/main	refs/heads/feat2	ignored	no baseline (no build for base branch ref)	no baseline coverage
refs/heads/feat1	not set	ignored	2	60%
refs/heads/feat1	refs/heads/main	ignored	1	50%
refs/pull/1/merge (PR from feat2 to main)	not set	true	1 (fallback to PR target branch)	50%
refs/pull/1/merge (PR from feat2 to main)	not set	false	no baseline (no build for refs/pull/1/merge yet)	no baseline coverage
refs/pull/1/merge (PR from feat2 to main)	refs/heads/main	ignored (because explicitly requested a baseline ref)	1	50%

Now, let's assume the pipeline for refs/pull/1/merge runs successful and has 80% code coverage and build ID 3. The next pipeline runs then look like this:

Branch of the build run	baseBranchRef	fallbackOnPRTargetBranch	Baseline build ID	Baseline coverage
refs/pull/1/merge (PR from feat2 to main)	not set	ignored (because build for current branch exists)	3 (previous run for refs/pull/1/merge)	80%
refs/pull/1/merge (PR from feat2 to main)	refs/heads/main	ignored (because explicitly requested a baseline ref)	1	50%
refs/pull/1/merge (PR from feat2 to main)	refs/heads/feat2	ignored (because explicitly requested a baseline ref)	no baseline (no build for base branch ref)	no baseline coverage

As you can see, the baseline logic is:

• If a previous successful run for current branch exists, take this as baseline.
• Else: if baseBranchRef is set and a previous successful run the that branch exists, take this as baseline.
• Else: if baseBranchRef is not set and running in PR and fallbackOnPRTargetBranch is set and a previous successful run for the PR target branch exists, take this as baseline.
• Else: no baseline

[QxBytes]

I have queried the API for the previous build and it shows total: 687, covered: 269 which is 39.15%, which shows me where we get that coverage percentage. However, how is this calculated? I can't find this number anywhere else in the logs in the previous pipeline run. I assumed that whatever was output in the BQC task log output as the current percentage would be the baseline percentage in the next given the previous build id is what I expect.

Also on the build summary page it merely shows

for build 120470974

I query the newer build as well and it also shows total: 687, covered: 269 (newer build is 120477700)-- but BQC reported it as 56.11%?).

My situation with the baseBranchRef I believe is straightforward:

• All my build runs are on the same branch (the feature branch)
• I am the only one on this branch
• The first build on that branch 120470974 runs and completes (succeeds)
• The second build on that branch 120477700 is run after with no changes (succeeds)
• Those are the two latest builds on the branch

[ReneSchumacher]

Since this seems to be a Microsoft internal project, I have added you to the internal communication. There might be an internal issue with Azure DevOps at the moment. PG is already engaged.