Skip to content

feat: add compare_estimators_tool for agentic model selection#370

Open
rupeshca007 wants to merge 1 commit into
sktime:mainfrom
rupeshca007:feat/compare-estimators-tool
Open

feat: add compare_estimators_tool for agentic model selection#370
rupeshca007 wants to merge 1 commit into
sktime:mainfrom
rupeshca007:feat/compare-estimators-tool

Conversation

@rupeshca007

Copy link
Copy Markdown
Contributor

Summary

Closes #369

What this adds

A new compare_estimators_tool MCP tool that enables agentic model selection directly from natural language prompts.

How it works

  1. LLM agent receives: "Find best forecasting model for this dataset"
    1. Agent calls compare_estimators(dataset="airline", estimator_names=[...])
    1. Tool runs expanding-window cross-validation on each model
    1. Returns a ranked leaderboard with the best model identified

Example output

{
  "success": true,
  "best_model": "NaiveForecaster",
  "best_score": 0.127075,
  "leaderboard": [
    {"rank": 1, "estimator": "NaiveForecaster", "test_MeanAbsolutePercentageError": 0.127075, "status": "ok"},
    {"rank": 2, "estimator": "ThetaForecaster", "test_MeanAbsolutePercentageError": 0.129265, "status": "ok"}
  ],
  "metric_used": "test_MeanAbsolutePercentageError",
  "cv_folds": 3,
  "dataset": "airline",
  "failed": []
}

Files changed

  • src/sktime_mcp/tools/compare_estimators.py - new tool (builds on evaluate_estimator_tool)
    • src/sktime_mcp/tools/__init__.py - registered compare_estimators_tool
    • src/sktime_mcp/server.py - registered tool in list_tools() and call_tool() dispatcher

- New compare_estimators_tool: evaluates multiple sktime forecasters
  on a dataset via expanding-window cross-validation and returns a
  ranked leaderboard with the best model identified. Core building
  block for agentic model selection workflows.
- Registered tool in server.py (list_tools + call_tool dispatcher)
  and 	ools/__init__.py
- Added missing horizon validation tests for it_predict_tool and
  it_predict_async_tool (closes sktime#367)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ENH] Add compare_estimators_tool for agentic model selection

1 participant