-
Notifications
You must be signed in to change notification settings - Fork 163
feat: model size status page #8604
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
The OLAPListTables API returns paginated results with a nextPageToken field. Previously only the first page was fetched, causing materialized models and tables on subsequent pages to show as having no size data. This change replaces the derived store pattern with a readable store that: - Fetches all pages sequentially for each connector - Detects nextPageToken in responses and creates follow-up queries - Accumulates all tables before building the final size map - Handles loading/error states correctly This fixes the issue where 5 models were detected but only 2 tables (from the first page) were returned, causing models like "auction_data_model" to show "-" instead of their actual size. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The issue was that useModelTableSizes was creating a new store on every render when resources changed. If the resources loaded before the queries completed, the component would be unmounted/remounted, cancelling the queries. This change: 1. Caches stores by instanceId:connectorArray to prevent recreation 2. Eagerly subscribes to each store to keep queries alive 3. Uses WeakMap to prevent memory leaks from old subscriptions 4. Ensures queries complete even if component re-renders quickly Now when you refresh the status page, queries stay alive in the background and complete asynchronously, updating the table when ready. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
|
a first pass at getting something in while project status 2.0 gets started as well as https://linear.app/rilldata/issue/PLAT-376/instrumentation-for-physical-table-size-of-models |
The store cache was keeping stale subscriptions alive across page refreshes. When you refreshed the page, the old cached store would be returned even if the resources had changed or needed fresh data. This change simplifies the approach: - Create fresh stores on each useModelTableSizes call - Let TanStack Query handle result caching (HTTP level) - Ensure queries always run with latest connector data - No issues with premature garbage collection Now on page refresh, sizes will load correctly immediately. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The issue was that columns were defined as a static const, so the accessor function captured the initial tableSizes reference and never updated when tableSizes changed. Changes: 1. Made columns reactive with $: so they update when tableSizes changes 2. Added console logging to ProjectResources and ProjectResourcesTable to track when stores and columns are updated 3. Added logging to selectors to see preload and store update timing This should fix the issue where the table shows "-" on initial load but correctly shows sizes after navigating away and back. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The core issue: columns were defined as a static const, capturing the initial tableSizes reference. When the store updated with new data, the column accessor functions still referenced the old empty Map. Solution: Make columns reactive with $: so they recreate whenever tableSizes changes, ensuring accessor functions get the current data. This allows the UI to update asynchronously as size data arrives. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
|
fixed bug, where values not shown unless action done on cloud, now it updates automatically: https://www.loom.com/share/825b70c51d234177a8949d7c920e4894 |
ericpgreen2
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for taking this on! I've been thinking about the design for this feature and want to propose a different direction before we proceed.
The current approach adds a "Size" column to the existing Resources table. While this works, I think it mixes two conceptually different concerns:
- Resources table = "Is my deployment healthy?" (reconciliation status, refresh times, errors)
- Table metrics = "How much data do I have?" (size, row count, column count)
This is why most rows show "-" for size — only Models map to database tables. Explores, Metrics Views, Canvas, etc. don't have a "size" concept.
Proposed Alternative: Separate "Tables" Section
I'd recommend mirroring the CLI's approach, where rill project status and rill project tables are distinct commands:
Resources
┌──────────────┬─────────────────────┬────────┬───────────────┬──────────────┐
│ Type │ Name │ Status │ Last refresh │ Next refresh │
├──────────────┼─────────────────────┼────────┼───────────────┼──────────────┤
│ Model │ bids_data_model │ ✓ │ Jan 12, 9:05 │ - │
│ Metrics View │ bids_metrics │ ✓ │ Jan 12, 9:05 │ - │
│ Explore │ bids_explore │ ✓ │ Jan 12, 9:05 │ - │
└──────────────┴─────────────────────┴────────┴───────────────┴──────────────┘
Tables
┌─────────────────────┬───────────┬─────────────┬───────────┐
│ Name │ Row Count │ Column Count│ Size │
├─────────────────────┼───────────┼─────────────┼───────────┤
│ bids_data_model │ 1,854,664 │ 31 │ 46.7 MiB │
│ auction_data_model │ 82,995 │ 9 │ 952.4 MiB │
└─────────────────────┴───────────┴─────────────┴───────────┘
Benefits:
- Cleaner separation of concerns (deployment health vs. data volume)
- No empty columns — every row in the Tables section has meaningful data
- Mirrors CLI experience (
rill project tablesoutput) - Room to add row count and column count (like CLI) without cluttering the Resources table
- Simpler data model — no need to join two APIs with matching logic
Trade-offs:
- More UI surface area (two tables instead of one)
- Users need to look in two places
What do you think? Happy to discuss if you see it differently or have concerns with this approach.
Specific code comments (if we proceed with current approach):
-
selectors.ts— The caching and preloading logic is quite complex (~200 lines). If we go with a separate Tables section, this could be simplified since we'd just callOLAPListTablesdirectly without needing to join with resources. -
ProjectResourcesTable.svelte— The{#key tableSizes}block will cause the entire table to re-render when sizes load. This might cause a jarring UX with the table flickering.
Developed in collaboration with Claude Code
|
I've got no strong opinions at the moment, thought it would be easier to use an existing space is all! And agreed, I've more or less completed the PRD to revamp our status pages, waiting for Di to return from holiday to get that one moving. Will re-iterate with a separate table! |
The row count fetching was failing with 401 errors because mutations weren't being triggered. Fixed by: - Explicitly calling .mutate() on row count mutations instead of just subscribing - Removing ProjectTablesRowCounts component with manual JWT waiting logic - Leveraging TanStack Query's built-in JWT handling via httpClient interceptor - Using mutation state subscription for proper success/error handling The key insight: createQueryServiceQuery returns a mutation store that requires explicit .mutate() calls to execute—just subscribing to it does nothing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
createQueryServiceQuery is a query creator, not a mutation creator. It needs: 1. Query parameters (instanceId, queryServiceQueryBody) as first argument 2. Query options with enabled: true as third argument 3. No .mutate() call - queries auto-execute when subscribed The query store will automatically handle JWT auth through httpClient interceptor. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Simplified row count state checking to match column count pattern - Added process flag to ensure counts only increment once - Added console logging to diagnose query state issues - Check for state.data?.data directly instead of state.isSuccess 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Log all state properties: isLoading, isFetching, isSuccess, isError, data, error, failureReason - Try multiple paths to find row data: direct array, nested .data, or .results - Better error handling to catch failureReason This will help identify where the actual row count data is stored in the query response. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Replace TanStack Query mutation wrapper with direct async httpClient function: - queryServiceQuery(instanceId, queryBody) returns a Promise - httpClient handles JWT auth automatically via interceptor - Simpler async/await pattern instead of store subscriptions - Removed complex state checking logic This approach is cleaner and properly leverages the built-in auth interceptor. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The 401 errors were happening because queryServiceQuery was running before the JWT token was available in the runtime store. Now we: - Import runtime store and get function - Wait for JWT token to be populated (up to 5 seconds) - Only make the query once JWT is ready This mirrors the pattern from ProjectTablesRowCounts.svelte but keeps the async/await approach integrated in the selector. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The 401 errors were happening because async row count fetching in the store setup was racing with JWT initialization. Now: 1. Export fetchRowCount(instanceId, tableName) from selectors.ts - Uses httpClient which has built-in JWT waiting via maybeWaitForFreshJWT - Returns Promise<number | 'error'> 2. Create ProjectTablesRowCounts.svelte component - Uses Svelte reactive statements ($:) to wait for JWT - Only fetches when: JWT ready AND tables available - Tracks fetched tables to avoid duplicates 3. Update ProjectTables.svelte - Import and use ProjectTablesRowCounts component - Pass rowCounts from component to ProjectTablesTable - Component-level approach guarantees JWT is ready This mirrors the original working pattern but uses httpClient which has proper JWT sequencing built in. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Instead of relying on httpClient's JWT handling, manually: - Get runtime store to access JWT token and host - Build full URL with host from runtime state - Add Authorization header with Bearer token directly - Add detailed logging for debugging 401 errors This eliminates any abstraction layers and makes JWT handling explicit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Manual JWT handling was causing 401 errors because the JWT in the store wasn't being properly validated or refreshed. Using httpClient instead leverages the built-in maybeWaitForFreshJWT interceptor which: - Waits for fresh JWT when needed - Handles token refresh automatically - Properly validates tokens with the backend This eliminates manual JWT token management. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
|
@ericpgreen2 are you aware of any issues with running query from Rill Cloud? im getting 401s and some other issues due to how rill project tables is designed. Row Count isn't the most important so we can remove it, too. LMK! |
|
note to self: https://github.com/rilldata/rill/actions/runs/20965177038/job/60253446299?pr=8604#step:14:865 |
|
Short term Solution: Remove row and column count from UI. For now, will remove the UI features. |

as discussed: https://rilldata.slack.com/archives/CTCJ58H3M/p1767734884155469
Checklist: