[bug] Fix batch INSERT_OVERWRITE replacecommits dropping adds in HudiDataFileExtractor#816
Open
vinishjail97 wants to merge 1 commit intoapache:mainfrom
Open
Conversation
…leExtractor When XTable processes multiple INSERT_OVERWRITE replacecommit instants in a single batch (e.g., A replaces initial, B replaces A on the same partition), getUpdatesToPartitionForReplaceCommit uses getReplacedFileGroupsBeforeOrOn(A.timestamp) which excludes A's file groups (replaced by B, and B > A). Combined with getAllFileGroups also excluding them (globally marked replaced), replacecommit A emits 0 adds — causing downstream integrity failures. Fix: change getReplacedFileGroupsBeforeOrOn -> getAllReplacedFileGroups in the replace commit handler, consistent with how getUpdatesToPartition already handles regular commits. The newFileIds/replacedFileIds sets from the commit's write stats already ensure only the correct files are added or removed. Added insertOverwrite() test helper to TestSparkHudiTable and an integration test testMultipleInsertOverwriteOnSamePartitions that reproduces and validates the fix.
vinishjail97
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When XTable processes multiple
INSERT_OVERWRITEreplacecommitinstants in a single batch (e.g., commit A replaces the initial file groups, then commit B replaces A's file groups on the same partition),getUpdatesToPartitionForReplaceCommitusesgetReplacedFileGroupsBeforeOrOn(A.timestamp)to find replaced file groups. This excludes A's own file groups because they were replaced by B (and B's timestamp > A's), andgetAllFileGroupsalso excludes them (globally marked replaced). The result: replacecommit A emits 0 adds, causing downstream format sync failures.Fix
Change
getReplacedFileGroupsBeforeOrOn→getAllReplacedFileGroupsin the replace commit handler. This is consistent with howgetUpdatesToPartitionalready handles regular commits (where all file groups across the timeline are considered). ThenewFileIdsandreplacedFileIdssets derived from each commit's write stats already ensure only the correct files are emitted as adds or removes for that specific commit.Tests
insertOverwrite()helper toTestSparkHudiTabletestMultipleInsertOverwriteOnSamePartitionsintegration test inITHudiConversionSourcethat reproduces the bug and validates the fixexpected: <[...parquet]> but was: <[]>) and passes with it