Multiple MarkdownExporters not possible

[zaharchenko-evgeny]

Multiple MarkdownExporters not possible

I'm testing BenchmarkDotNet and would like to have the benchmark results exported to both GitHub markdown and Atlassian markdown. But I get the error

The exporter BenchmarkDotNet.Exporters.MarkdownExporter is already present in configuration. There may be unexpected results.

Is that by intention - and why is it not possible to export to both markdown formats in the same run?

Hints

adamsitnik: > Is that by intention

I really doubt that, it's most likely a bug on our side.

bstordrup: The issue is as far as I can see in the ImmutableConfigBuilder.GetExporters method. It makes a mergeDictionary by the System.Type - and that will be the same for both the GitHub MarkdownExporter and the Atlassian MarkdownExporter.

If the purpose is to avoid having for example GitHub MarkdownExporter twice in the merged list, I see the logic in going through the lists of exporters and remove duplicates. But it has to be on something else or more that just the System.Type.

[dpaia-validator]

✅ Datapoint verification passed.

Instance: dotnet__benchmarkdotnet-2395
Duration: 104s
Tests: Total: 897, Passed: 897, Failed: 0, Skipped: 0
FAIL_TO_PASS: pass: all fail_to_pass tests fixed

Details: Workflow run

[dpaia-validator]

Datapoint generated successfully.

Instance: dotnet__benchmarkdotnet-2395
Dataset PR: https://github.com/dpaia/dataset/pull/1160

[dpaia-validator]

Datapoint pipeline complete. Closing source PR (not merging).

[dpaia-validator]

✅ Datapoint verification passed.

Instance: dotnet__benchmarkdotnet-2395
Duration: 53s
Criteria: compilation: pass, baseline_tests: pass, patch_applied: pass, tests: pass, fail_to_pass: pass, pass_to_pass: pass
Tests: Total: 897, Passed: 897, Failed: 0, Skipped: 0
FAIL_TO_PASS: pass: all fail_to_pass tests fixed

Details: Workflow run

[dpaia-validator]

Datapoint generated successfully.

Instance: dotnet__benchmarkdotnet-2395
Dataset PR: https://github.com/dpaia/dataset/pull/1271

[dpaia-validator]

Datapoint pipeline complete. Closing source PR (not merging).

[dpaia-validator]

❌ Datapoint verification failed.

Instance: dotnet__benchmarkdotnet-2395
Duration: 35s
Criteria: compilation: pass, baseline_tests: pass, patch_applied: pass, tests: pass, fail_to_pass: fail, pass_to_pass: fail
Tests: Total: 0, Passed: 0, Failed: 0, Skipped: 0
FAIL_TO_PASS: fail: eval missing: MultipleExportersOfSameTypeWithDifferentNamesAreAccepted

Failed criteria: fail_to_pass: fail, pass_to_pass: fail

Error details

fail_to_pass: eval missing: MultipleExportersOfSameTypeWithDifferentNamesAreAccepted
pass_to_pass: eval missing: AddOneVariable, AddVariableArray, AddVariableArrayWithDuplications, AllDifferences, AllEnvRequirementsAreSupported, AllJobModesPropertyNamesMatchCharacteristicNames, AllMutatorsAttributesCanBeAppliedOnlyOncePerType, AllRuntimeMonikerAreKnown, AlphaNumericComparisionTest, AmdIsPrettifiedWithDiffFrequencies(originalName: "AMD Ryzen 7 2700X Eight-Core Processor", prettifiedName: "AMD Ryzen 7 2700X 4.10GHz", actualFrequency: 4.0999999999999996, physicalCoreCount: 8, logicalCoreCount: 16)

Details: Workflow run

[dpaia-validator]

✅ Datapoint verification passed.

Instance: dotnet__benchmarkdotnet-2395
Duration: 54s
Criteria: compilation: pass, baseline_tests: pass, patch_applied: pass, tests: pass, fail_to_pass: pass, pass_to_pass: pass
Tests: Total: 897, Passed: 897, Failed: 0, Skipped: 0
FAIL_TO_PASS: pass: all fail_to_pass tests fixed

Details: Workflow run

[zaharchenko-evgeny]

@dpaia-validator generate datapoint

[dpaia-validator]

Datapoint generated successfully.

Instance: dotnet__benchmarkdotnet-2395
Dataset PR: https://github.com/dpaia/dataset/pull/1331

[dpaia-validator]

Datapoint pipeline complete. Closing source PR (not merging).

[dpaia-validator]

❌ Datapoint verification failed.

Instance: dotnet__benchmarkdotnet-2395
Duration: (unknown)s
Criteria: (unavailable)

Error details

        "RunningBenchmarksWithIncompatibleConfigsMustFailWithCriticalError",
        "SpecifyingInvalidStatisticalTestsThresholdMeansFailure",
        "SummaryWithFailureReportDoesNotThrowNre",
        "TooManyHardwareCounterNameMeansFailure",
        "UnknownConfigMeansFailure",
        "UnknownMSBuildErrorsAreNotTranslatedToMoreUserFriendlyVersions",
      "criterion": "fail_to_fail",
      "detail": "no expected fail_to_fail tests",
Validation failed (1 failed criteria, status=failure)
  FAIL: pass_to_pass — eval missing: MaxParameterColumnWidthCanBeCustomized(initialLongText: ""···, maxParameterColumnWidth: 20, expectedDisplayText: "(...) [54]"), MaxParameterColumnWidthCanBeCustomized(initialLongText: ""···, maxParameterColumnWidth: 20, expectedDisplayText: "(...)b [55]"), MaxParameterColumnWidthCanBeCustomized(initialLongText: "a�"···, maxParameterColumnWidth: 20, expectedDisplayText: "a(...) [55]"), MaxParameterColumnWidthCanBeCustomized(initialLongText: "a�"···, maxParameterColumnWidth: 20, expectedDisplayText: "a(...)b [56]"), TryGetVersionFromRuntimeDirectoryHandlesInvalidInput(runtimeDirectory: "/mnt/agent/temp/buildTmp/publish"), TryGetVersionFromRuntimeDirectoryHandlesInvalidInput(runtimeDirectory: "/mnt/agent/temp/buildTmp/publish/"), TryGetVersionFromRuntimeDirectoryHandlesValidInput(runtimeDirectory: "/mnt/agent/temp/buildTmp/123.0.0/", expectedTfm: NotRecognized, expectedMsBuildMoniker: "net123.0"), TryGetVersionFromRuntimeDirectoryHandlesValidInput(runtimeDirectory: "/mnt/agent/temp/buildTmp/2.0.9/", expectedTfm: NetCoreApp20, expectedMsBuildMoniker: "netcoreapp2.0"), TryGetVersionFromRuntimeDirectoryHandlesValidInput(runtimeDirectory: "/mnt/agent/temp/buildTmp/2.1.12/", expectedTfm: NetCoreApp21, expectedMsBuildMoniker: "netcoreapp2.1"), TryGetVersionFromRuntimeDirectoryHandlesValidInput(runtimeDirectory: "/mnt/agent/temp/buildTmp/2.2.6/", expectedTfm: NetCoreApp22, expectedMsBuildMoniker: "netcoreapp2.2")

Details: Workflow run

[dpaia-validator]

✅ Datapoint verification passed.

Instance: dotnet__benchmarkdotnet-2395
Duration: 50s
Criteria: compilation: pass, baseline_tests: pass, patch_applied: pass, tests: pass, fail_to_pass: pass, pass_to_pass: pass, fail_to_fail: skipped
Tests: Total: 897, Passed: 897, Failed: 0, Skipped: 0
FAIL_TO_PASS: pass: all fail_to_pass tests fixed

Details: Workflow run

[dpaia-validator]

✅ Datapoint verification passed.

Instance: dotnet__benchmarkdotnet-2395
Duration: 48s
Criteria: compilation: pass, baseline_tests: pass, patch_applied: pass, tests: pass, fail_to_pass: pass, pass_to_pass: pass, fail_to_fail: skipped
Tests: Total: 897, Passed: 889, Failed: 0, Skipped: 8
FAIL_TO_PASS: pass: all fail_to_pass tests fixed

Details: Workflow run

[dpaia-validator]

✅ Datapoint verification passed.

Instance: dotnet__benchmarkdotnet-2395
Duration: 48s
Criteria: compilation: pass, baseline_tests: pass, patch_applied: pass, tests: pass, fail_to_pass: pass, pass_to_pass: pass, fail_to_fail: skipped
Tests: Total: 897, Passed: 889, Failed: 0, Skipped: 8
FAIL_TO_PASS: pass: all fail_to_pass tests fixed

Details: Workflow run

[dpaia-validator]

Datapoint generated successfully.

Instance: dotnet__benchmarkdotnet-2395
Dataset PR: https://github.com/dpaia/dataset/pull/1342