[Core][Bug-fix] Fix wrong merge order of increment diff split read #7033
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…write of a key with non-deterministic sequence number.
Author
|
@JingsongLi Please help review this MR. Thx! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
This pull request aims to fix an issue in the incremental scan where concurrent writes to the same key could finally return wrong merged result. Since records included in merging set are in non-deterministic sequence numbers could result in unstable merge order, which the incremental scan heavily relies on, would lead to incorrect final results. The fix ensures the correct result by re-sorting the data in the getResult stage of the split diff read process.
Tests
A new integration test, testIncrementScanModeWithInsertOverwrite, has been added to BatchFileStoreITCase.java. This test simulates the scenario with multiple INSERT OVERWRITE operations with different seq. number on the same key, and then verifies that the diff mode of incremental scan correctly identifies the changes, ensuring the fix is effective.
API and Format
No API or format changes. This change only involves an internal logic correction and does not affect any external APIs, data storage formats, or configuration files.
Documentation
This change is an internal bug fix and does not require updates to user-facing documentation.