chore: switch from sonnet to haiku in maintainers processing [CM-1049]#3915
chore: switch from sonnet to haiku in maintainers processing [CM-1049]#3915
Conversation
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
|
Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability. Example:
Projects:
Please add a Jira issue key to your PR title. |
|
|
|
Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability. Example:
Projects:
Please add a Jira issue key to your PR title. |
|
Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability. Example:
Projects:
Please add a Jira issue key to your PR title. |
There was a problem hiding this comment.
Pull request overview
This PR updates the Git Integration maintainer-extraction pipeline to use a different AWS Bedrock Claude model, adjusts the extraction prompt guidance, and updates response parsing/cost calculation to better match the new model’s behavior and pricing.
Changes:
- Switched Bedrock inference from Claude Sonnet 4 to Claude Haiku 4.5 and updated token cost calculations accordingly.
- Added markdown code-fence stripping to handle cases where the model wraps JSON in ``` blocks.
- Refined the maintainer/contributor extraction prompt rules (role/title derivation, filename-to-normalized_title mapping, and inclusion rules for people without emails).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| services/apps/git_integration/src/crowdgit/services/maintainer/maintainer_service.py | Tightens the LLM extraction prompt to improve role/title normalization and ensure people aren’t omitted when email is missing. |
| services/apps/git_integration/src/crowdgit/services/maintainer/bedrock.py | Switches Bedrock model + updates cost calculation and adds response post-processing for markdown-wrapped JSON outputs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
| if raw_text.startswith("```"): | ||
| raw_text = raw_text.split("\n", 1)[-1] | ||
| if raw_text.endswith("```"): | ||
| raw_text = raw_text.rsplit("```", 1)[0] | ||
| raw_text = raw_text.strip() |
There was a problem hiding this comment.
The markdown code-fence stripping can still leave a trailing closing fence in common outputs like json\n{...}\n\n (note the newline after the closing fence). In that case raw_text.endswith("```") is false before .strip(), so the closing fence remains and json.loads(raw_text) will fail. Consider stripping whitespace before checking for the closing fence (or using a small regex that removes leading .*\n and trailing with surrounding whitespace).
| if raw_text.startswith("```"): | |
| raw_text = raw_text.split("\n", 1)[-1] | |
| if raw_text.endswith("```"): | |
| raw_text = raw_text.rsplit("```", 1)[0] | |
| raw_text = raw_text.strip() | |
| stripped_text = raw_text.strip() | |
| if stripped_text.startswith("```"): | |
| # Remove the opening code fence (and optional language specifier) | |
| stripped_text = stripped_text.split("\n", 1)[-1] | |
| # Tolerate trailing whitespace/newlines after the closing fence | |
| if stripped_text.rstrip().endswith("```"): | |
| stripped_text = stripped_text.rstrip().rsplit("```", 1)[0] | |
| raw_text = stripped_text.strip() |
| try: | ||
| body_bytes = await response["body"].read() | ||
| response_body = json.loads(body_bytes.decode("utf-8")) | ||
| raw_text = response_body["content"][0]["text"].replace('"""', "").strip() |
There was a problem hiding this comment.
If decoding/parsing the Bedrock response body fails at response_body = json.loads(...), the except block later will try to log response_body["content"][0]["text"], but response_body will be undefined, masking the original error. Consider initializing response_body/raw_text to a safe default before the try, and in the exception path log the raw body_bytes (or decoded text) when JSON parsing fails.
| # Strip markdown code fences if present (Haiku sometimes ignores the system prompt) | ||
| if raw_text.startswith("```"): | ||
| raw_text = raw_text.split("\n", 1)[-1] | ||
| if raw_text.endswith("```"): | ||
| raw_text = raw_text.rsplit("```", 1)[0] | ||
| raw_text = raw_text.strip() | ||
|
|
||
| output = json.loads(raw_text) |
There was a problem hiding this comment.
The new parsing behavior (stripping markdown code fences before json.loads) is a regression-prone edge case that would benefit from a unit test (e.g., outputs with ```json fences, trailing newlines, and plain JSON). There is an existing pytest suite under services/apps/git_integration/src/test, but no coverage around this Bedrock response parsing currently.
mbani01
changed the title
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
| input_cost = (input_tokens / 1000) * 0.003 | ||
| output_cost = (output_tokens / 1000) * 0.015 | ||
| input_cost = (input_tokens / 1_000_000) * 0.80 | ||
| output_cost = (output_tokens / 1_000_000) * 4.00 |
There was a problem hiding this comment.
Incorrect Haiku 4.5 token pricing underestimates costs
Medium Severity
The cost calculation uses $0.80 per 1M input tokens and $4.00 per 1M output tokens, but the actual AWS Bedrock pricing for Claude Haiku 4.5 is $1.00 per 1M input tokens and $5.00 per 1M output tokens. Both the comment and the multiplier values are wrong, causing cost tracking to underreport actual spend by 20%.
| input_cost = (input_tokens / 1000) * 0.003 | ||
| output_cost = (output_tokens / 1000) * 0.015 | ||
| input_cost = (input_tokens / 1_000_000) * 0.80 | ||
| output_cost = (output_tokens / 1_000_000) * 4.00 |
This pull request updates the Bedrock model used for maintainer extraction, improves handling of model output, updates cost calculations, and clarifies the extraction prompt for maintainers and contributors. The most important changes are grouped below:
Bedrock Model and Output Handling:
claude-sonnet-4toclaude-haiku-4.5for faster and potentially more cost-effective inference (bedrock.py).Cost Calculation:
claude-haiku-4.5pricing ($0.80 per 1M input tokens, $4.00 per 1M output tokens), replacing the previous Sonnet 4 rates.Maintainer Extraction Prompt Improvements:
normalized_titlebased on filename patterns, making the mapping from filename to role more explicit and robust.Note
Medium Risk
Medium risk because it changes the underlying LLM model and parsing behavior for maintainer extraction, which can alter extracted results and downstream maintainer updates/cost reporting.
Overview
Switches the Bedrock invocation used for maintainer extraction from Claude Sonnet to Claude Haiku 4.5, and updates token cost calculation to Haiku’s per-1M-token pricing.
Hardens response handling by stripping optional markdown code fences before
json.loads()to tolerate non-compliant model output.Refines the maintainer extraction prompt to derive consistent per-person titles when roles aren’t explicitly labeled, to map
normalized_titlemore deterministically from filename patterns, and to require including people even when emails are missing.Written by Cursor Bugbot for commit 1dea030. This will update automatically on new commits. Configure here.