Skip to content

feat(llm): add configurable GenerationParams across agent runners#319

Open
Sagar-CK wants to merge 1 commit into
IBM:mainfrom
Sagar-CK:feature/configurable-generation-params
Open

feat(llm): add configurable GenerationParams across agent runners#319
Sagar-CK wants to merge 1 commit into
IBM:mainfrom
Sagar-CK:feature/configurable-generation-params

Conversation

@Sagar-CK
Copy link
Copy Markdown

Description

Standardizes LLM generation configuration across all agent runners and the LiteLLM backend. Replaces hardcoded values (e.g. max_tokens=2048) with a shared, env-driven GenerationParams dataclass and per-SDK mappers.

Problem

  • LiteLLMBackend hardcoded max_tokens=2048 with no override path
  • No way to enable/disable extended thinking / reasoning consistently
  • temperature was the only tunable param; no shared place for top_p, stop, etc.
  • SDK runners (claude-agent, openai-agent, deep-agent) had no shared generation config

Solution

  • Introduce GenerationParams (src/llm/generation.py) with env defaults via LLM_* vars
  • Thread through LLMBackend.generate(..., *, params=) with merge semantics (env → ctor → per-call; temperature= always wins)
  • Map to each SDK: LiteLLM kwargs, ClaudeAgentOptions (effort/thinking), OpenAI ModelSettings, LangChain ChatOpenAI
  • Gate reasoning by model family heuristics; strip + warn when unsupported (e.g. WatsonX)
  • Add unit tests for env parsing, merge semantics, and litellm kwargs matrix

Env vars (documented in .env.public):

  • LLM_MAX_TOKENS (default: 4096)
  • LLM_TEMPERATURE (default: 0.0)
  • LLM_TOP_P, LLM_STOP, LLM_REASONING_EFFORT, LLM_THINKING_BUDGET_TOKENS

Type of Change

  • New Benchmark Scenario (Industry/Asset type)
  • Evaluation Metric / Scorer
  • Agentic Orchestration Logic (ReAct, Plan-Execute, etc.)
  • Infrastructure / Tooling Improvement

Related Issues

Testing & Validation

  • Unit Tests: uv run pytest src/ -v -k "not integration" — 357 passed

@DhavalRepo18
Copy link
Copy Markdown
Collaborator

@Sagar-CK It seems that too many changes are coming along.

This is potentially due to formatting. Will it be okay if you disable formatting for this PR, and maybe we can make a pass to reformat all the files with standard formatting?

Copy link
Copy Markdown
Collaborator

@DhavalRepo18 DhavalRepo18 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Submit code without Python code formatting.

@DhavalRepo18
Copy link
Copy Markdown
Collaborator

@Sagar-CK I left few comment.

Signed-off-by: Sagar Chethan Kumar <sagar.chethankumar@gmail.com>
@Sagar-CK Sagar-CK force-pushed the feature/configurable-generation-params branch from 5533df0 to 98d2115 Compare May 24, 2026 19:17
@Sagar-CK
Copy link
Copy Markdown
Author

@DhavalRepo18 have updated the PR, tests still pass, let me know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants