fix: enable prompt caching for AWS Bedrock custom ARN models#11984
Open
roomote-v0[bot] wants to merge 1 commit intomainfrom
Open
fix: enable prompt caching for AWS Bedrock custom ARN models#11984roomote-v0[bot] wants to merge 1 commit intomainfrom
roomote-v0[bot] wants to merge 1 commit intomainfrom
Conversation
Add cachableFields to guessModelInfoFromId() for Claude model patterns so that custom ARN models matching known Claude patterns get proper caching metadata. Update supportsAwsPromptCache() to respect the user's explicit opt-in for custom ARN models, even when the model ID from the ARN is not recognized. When the Bedrock API receives cache points for a model that does not support caching, it simply ignores them without error. Fixes #11983
|
@roomote I believe you can move this pull request out of draft status - it looks ready to go |
Contributor
Author
Moved this PR out of draft status -- it's now ready for review. |
roomote-v0
bot
requested review from
cte,
daniel-lxs,
hannesrudolph,
jr and
mrubens
as code owners
dosubot
bot
added
size:L
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related GitHub Issue
Closes: #11983
Description
This PR attempts to address Issue #11983 where prompt caching does not work with AWS Bedrock custom ARN models. Feedback and guidance are welcome.
Root cause: When a custom ARN is used and the model ID extracted from the ARN is not found in
bedrockModels,guessModelInfoFromId()returns model info withoutcachableFields. ThesupportsAwsPromptCache()method requires bothsupportsPromptCache: trueANDcachableFields.length > 0, so prompt caching is silently disabled even though the user has explicitly enabled it.Changes made in
src/api/providers/bedrock.ts:Added
cachableFieldstoguessModelInfoFromId()- For each Claude model pattern that setssupportsPromptCache: true, the correspondingcachableFields: ["system", "messages", "tools"]is now also included. This ensures that when a custom ARN's model ID is guessed via pattern matching, the full caching metadata is present.Updated
supportsAwsPromptCache()to respect user intent for custom ARN models - When a custom ARN is in use and the user has not explicitly disabled prompt caching (awsUsePromptCache !== false), the method now injects defaultcachableFieldsand enables caching. This is safe because the Bedrock API simply ignores cache points for models that do not support caching, without erroring.Test Procedure
src/api/providers/__tests__/bedrock.spec.ts:awsUsePromptCache: falsedisables cachingguessModelInfoFromIdincludescachableFieldsfor Claude patternsPre-Submission Checklist
Interactively review PR in Roo Code Cloud