feat: implement unlimited pagination for message browsing #2128
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add PageToken struct with encode/decode, validation, and cursor management for multi-partition offset tracking. Supports both ascending and descending order pagination.
Add page_token, page_size, next_page_token, and has_more fields to support cursor-based pagination in ListMessages API.
- Add PageToken and PageSize to ListMessagesRequest - Pass next_page_token and has_more in completion message - Validate pagination mode (maxResults=-1) and reject filters
- Add calculateConsumeRequestsWithPageToken with round-robin message distribution - Support both ascending and descending order pagination - Reverse messages per-partition for descending order - Fix offset boundary check (inclusive EndOffset) - Track finished partitions to prevent over-consumption
- Test first page and multi-page pagination - Verify descending order within partitions - Test filter rejection with pagination mode - Verify legacy mode compatibility
- Add pageToken/nextPageToken to message search state - Auto-load next page when user reaches last loaded page - Reset pagination when navigating back to page 1 - Disable sorting in unlimited mode (maxResults=-1) - Hide unlimited option in Latest/Live mode - Auto-reset to 50 when switching to Latest/Live from unlimited
…eter Remove hasMore as an explicit parameter from IListMessagesProgress.OnComplete(). The hasMore field is now automatically derived from whether nextPageToken is empty. This simplifies the interface by removing redundant state - the information about whether more pages exist is already encoded in the presence/absence of nextPageToken. Changes: - Remove hasMore parameter from OnComplete() interface (4 params -> 3 params) - Derive HasMore automatically: HasMore = nextPageToken != "" - Update all call sites to pass 3 parameters instead of 4 - Update integration tests to check nextPageToken instead of hasMore - Update mocks to match new 3-parameter signature The protobuf has_more field is still populated for wire compatibility, but is no longer passed as a separate parameter through the application layers.
- Fix mock expectations in integration tests to use 3 parameters - Remove HasMore field from stream completion message struct - Fix page token test to handle unordered partition iteration
Contributor
|
The latest Buf updates on your PR. Results from workflow Buf CI / validate (pull_request).
|
Frontend fixes: - Fix stale closure race condition in auto-load effect by capturing messageSearch reference and validating before state update - Add proper cleanup for loadMore operations on component unmount - Fix circular effect dependency in page reset by using ref instead of state in dependency array - Add retry limit (max 3 attempts) for loadMore failures to prevent infinite retry loops on network errors - Track mounted state to prevent state updates on unmounted component Backend fix: - Change pagination logging from INFO to DEBUG level to reduce log verbosity in production
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Implements unlimited pagination for Kafka message browsing, allowing users to navigate through arbitrarily large topics without loading all messages into memory.
Changes
Backend
ListMessagesproto (next_page_token)Frontend
Key Features