Smart Grid Transformer Integration & Automated Scenario Generation Pipeline#288
Smart Grid Transformer Integration & Automated Scenario Generation Pipeline#288Rohith-Kanathur wants to merge 30 commits into
Conversation
…ing, and terminal output refinements
…ine for benchmark scenario creation Introduces a fully automated 4-phase scenario generation pipeline driven by LiteLLM: PHASE 1 — Asset Profile Construction - LLM generates targeted ArXiv search queries from the asset's canonical academic name - Fetches PDFs via ArXiv API (up to 2 per query, first 5 pages extracted via pypdf) - Synthesises sensor mappings, failure modes, ISO standards, and relevant tool mappings into an AssetProfile (Pydantic model); fatal if unparseable PHASE 2 — Scenario Budget Allocation - LLM dynamically distributes the total scenario count across 5 subagent categories (iot, fmsr, tsfm, wo, multiagent) based on the AssetProfile - Multiagent capped at 50% of total; fatal if allocation is unparseable PHASE 3 — Individual Agent Generation & Validation (iot / fmsr / tsfm / wo) - Fetches 2 typed few-shot examples from ibm-research/AssetOpsBench on HuggingFace - SCENARIO_GENERATOR_PROMPT produces raw scenario dicts per subagent - VALIDATE_REPAIR_PROMPT corrects schema, tool alignment, and characteristic_form quality - Validation diffs (before/after) written to numbered log files when --log is active PHASE 4 — Multi-Agent Combiner - MULTIAGENT_COMBINER_PROMPT seeds from up to 10 single-agent scenarios to produce complex cross-subagent orchestration scenarios (e.g. IoT → FMSR → WO) CLI (python -m scenarios.generator): --num-scenarios N Total scenarios to generate (default: 50) --output PATH Output JSON path (default: generated_scenarios.json) --model-id MODEL LiteLLM model override --show-workflow Granular phase-by-phase terminal output with diffs --log Dump all raw prompts + responses to logs/<asset>_<ts>/ Supporting additions: - models.py: AssetProfile, ScenarioBudget, Scenario Pydantic models - prompts.py: 6 prompt templates (PROFILE_BUILDER, SCENARIO_GENERATOR, VALIDATE_REPAIR, MULTIAGENT_COMBINER, RESEARCH_QUERY_GENERATOR, BUDGET_ALLOCATOR) - utils.py: fetch_arxiv_studies() with multi-query dedup + PDF extraction; fetch_hf_fewshot() with type-filtered HuggingFace loading + mock fallback - Log header includes ArXiv paper titles and PDF URLs for full traceability - src/scenarios/README.md: full usage docs, pipeline breakdown, output schema, troubleshooting table, and log file structure reference
This reverts commit 64f0ae6.
|
Hi @Sagar-CK @Rohith-Kanathur I would like to know why only 5 data points are in the bulk_docs_transformer.json |
|
Hi @ShuxinLin The current bulk_docs_transformer.json only contains 5 data points because it was initially added as lightweight mock/test data to verify if the CouchDB ingestion was happening correctly. The intent wasn’t to model a full production level smart grid dataset but rather to validate the loader + CouchDB integration flow first. We can definitely expand the dataset with additional transformer documents. I was thinking about using this dataset since it contains around 400 transformer data points: https://data.mendeley.com/datasets/rz75w3fkxy/1 Please let me know what you think. Thank you. |
|
@Rohith-Kanathur You needed these sample data for what? These PR is for generating scenarios? or testing Scenario? |
|
@DhavalRepo18 I was talking about this github issue: #303 |
Overview
This PR delivers two major contributions to AssetOpsBench:
ScenarioGeneratorAgent) that scales benchmark creation to new asset classes without manual authoring1. Smart Grid Transformer Integration
CouchDB Data
Four New FMSR Tools
predict_health_indexinterpret_dgaassess_winding_temperatureassess_load_profileTests
2. Scenario Generation Pipeline (
ScenarioGeneratorAgent)A three-phase automated pipeline that generates physically plausible, causally consistent, and tool-reachable benchmark scenarios for any onboarded asset class.
Phase 1: Asset Profiling
Discovers live asset instances, sensors, and failure mappings from CouchDB, retrieves and synthesizes domain literature from ArXiv or Semantic Scholar, and merges everything with MCP tool schemas into a single structured
AssetProfilethat grounds all downstream generation.Phase 2: Budget Allocation
Distributes the total scenario count across focusses (
iot,fmsr,tsfm,wo,vibration,multiagent) proportionally to the asset's available data modalities and tool coverage, withmultiagentcapped at 75% of total budget to preserve lane diversity.Phase 3: Scenario Generation & Validation
Generates per-focus scenarios conditioned on the asset profile, runs each candidate through an LLM-based repair step.
Output
Each run produces a timestamped directory with
scenarios.json. Each scenario object contains anid,type,text,category, andcharacteristic_form.CLI