Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 69 additions & 0 deletions .github/workflows/conformance.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
name: Conformance Test

on:
pull_request:

permissions:
contents: read

jobs:
conformance:
runs-on: ubuntu-latest

steps:
- name: Check out code
uses: actions/checkout@v6
with:
# Fetch full history to access merge-base
fetch-depth: 0

- name: Set up Go
uses: actions/setup-go@v6
with:
go-version-file: "go.mod"

- name: Download dependencies
run: go mod download

- name: Run conformance test
id: conformance
run: |
# Run conformance test, capture stdout for summary
script/conformance-test > conformance-summary.txt 2>&1 || true

# Output the summary
cat conformance-summary.txt

# Check result
if grep -q "RESULT: ALL TESTS PASSED" conformance-summary.txt; then
echo "status=passed" >> $GITHUB_OUTPUT
else
echo "status=differences" >> $GITHUB_OUTPUT
fi

- name: Generate Job Summary
run: |
# Add the full markdown report to the job summary
echo "# MCP Server Conformance Report" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "Comparing PR branch against merge-base with \`origin/main\`" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY

# Extract and append the report content (skip the header since we added our own)
tail -n +5 conformance-report/CONFORMANCE_REPORT.md >> $GITHUB_STEP_SUMMARY

echo "" >> $GITHUB_STEP_SUMMARY
echo "---" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY

# Add interpretation note
if [ "${{ steps.conformance.outputs.status }}" = "passed" ]; then
echo "✅ **All conformance tests passed** - No behavioral differences detected." >> $GITHUB_STEP_SUMMARY
else
echo "⚠️ **Differences detected** - Review the diffs above to ensure changes are intentional." >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
echo "Common expected differences:" >> $GITHUB_STEP_SUMMARY
echo "- New tools/toolsets added" >> $GITHUB_STEP_SUMMARY
echo "- Tool descriptions updated" >> $GITHUB_STEP_SUMMARY
echo "- Capability changes (intentional improvements)" >> $GITHUB_STEP_SUMMARY
fi
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -19,4 +19,5 @@ bin/
# binary
github-mcp-server

.history
.history
conformance-report/
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -384,6 +384,7 @@ You can also configure specific tools using the `--tools` flag. Tools can be use
- Tools, toolsets, and dynamic toolsets can all be used together
- Read-only mode takes priority: write tools are skipped if `--read-only` is set, even if explicitly requested via `--tools`
- Tool names must match exactly (e.g., `get_file_contents`, not `getFileContents`). Invalid tool names will cause the server to fail at startup with an error message
- When tools are renamed, old names are preserved as aliases for backward compatibility. See [Deprecated Tool Aliases](docs/deprecated-tool-aliases.md) for details.

### Using Toolsets With Docker

Expand Down Expand Up @@ -459,7 +460,6 @@ The following sets of tools are available:
| `code_security` | Code security related tools, such as GitHub Code Scanning |
| `dependabot` | Dependabot tools |
| `discussions` | GitHub Discussions related tools |
| `experiments` | Experimental features that are not considered stable yet |
| `gists` | GitHub Gist related tools |
| `git` | GitHub Git API related tools for low-level Git operations |
| `issues` | GitHub Issues related tools |
Expand Down
Loading