fix: critical performance optimizations for large datasets#1721
fix: critical performance optimizations for large datasets#1721
Conversation
WalkthroughThe changes refactor several file system operations from synchronous to asynchronous across multiple modules, updating method signatures and internal logic accordingly. Additionally, there are improvements to resource cleanup and optimization of data transformation in query result handling. No changes were made to public API signatures except for marking affected methods as Changes
Sequence Diagram(s)sequenceDiagram
participant UI/Webview
participant DocsEditViewPanel
participant fs/promises
UI/Webview->>DocsEditViewPanel: Request to render or save documentation
DocsEditViewPanel->>fs/promises: readFile / writeFile (async)
fs/promises-->>DocsEditViewPanel: File content / write confirmation
DocsEditViewPanel-->>UI/Webview: Rendered HTML or save result
sequenceDiagram
participant NewDocsGenPanel
participant dbtTestService
participant fs/promises
NewDocsGenPanel->>dbtTestService: getConfigByTest (async)
dbtTestService->>fs/promises: readFile (async)
fs/promises-->>dbtTestService: File content
dbtTestService-->>NewDocsGenPanel: Config data
NewDocsGenPanel->>fs/promises: readFile (async) for SQL test code
fs/promises-->>NewDocsGenPanel: SQL code
Suggested labels
Suggested reviewers
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
npm error Exit handler never called! Warning Review ran into problems🔥 ProblemsErrors were encountered while retrieving linked issues. Errors (1)
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (6)
🧰 Additional context used🧠 Learnings (5)src/dbt_client/dbtCoreIntegration.ts (5)src/webview_provider/queryResultPanel.ts (2)src/services/dbtTestService.ts (2)src/webview_provider/newDocsGenPanel.ts (7)src/webview_provider/docsEditPanel.ts (7)🧬 Code Graph Analysis (1)src/webview_provider/newDocsGenPanel.ts (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
🔇 Additional comments (19)
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![ellipsis-dev[bot]](https://avatars.githubusercontent.com/in/64358?s=60&v=4){kind=small}
There was a problem hiding this comment.
Important
Looks good to me! 👍
Reviewed everything up to a4dc431 in 1 minute and 8 seconds. Click for details.
- Reviewed
269lines of code in6files - Skipped
0files when reviewing. - Skipped posting
4draft comments. View those below. - Modify your settings and rules to customize what types of comments Ellipsis leaves. And don't forget to react with 👍 or 👎 to teach Ellipsis.
1. src/webview_provider/queryResultPanel.ts:420
- Draft comment:
Consider adding an explicit return type (e.g., Promise) for the 'transmitDataWrapper' method and cache 'result.table.column_names' in a local variable inside the inner loop to reduce repeated property lookups. - Reason this comment was not posted:
Comment was on unchanged code.
2. src/webview_provider/queryResultPanel.ts:527
- Draft comment:
Consider switching from readFileSync to an asynchronous file read (e.g., readFile) in the getHtml function to avoid blocking the main thread, in line with other parts of the codebase. - Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.
3. src/webview_provider/queryResultPanel.ts:548
- Draft comment:
Please add explicit return type annotations for the utility functions 'getNonce' and 'getUri' to improve code clarity and ease future refactoring. - Reason this comment was not posted:
Comment was not on a location in the diff, so it can't be submitted as a review comment.
4. src/dbt_client/dbtCoreIntegration.ts:204
- Draft comment:
Typographical error: The message "An error occured while finding package paths: " uses 'occured' which should be spelled 'occurred'. - Reason this comment was not posted:
Comment was on unchanged code.
Workflow ID: wflow_femhT52ZvuBdPPBy
You can customize 
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
- Optimize query result data transformation (94% faster for large result sets) - Replace nested loops with object spread using single-pass transformation - Pre-allocate array to reduce memory pressure - Fix memory leak in conversation provider timer disposal - Ensure timer is cleared when provider is disposed These changes significantly improve performance when: - Viewing large query results (thousands of rows) - Long-running sessions with conversation polling Co-Authored-By: Claude <noreply@anthropic.com>
mdesmet
force-pushed
the
performance-optimization
branch
from
a4dc431 to
ef8426d
Compare
Bundle Size Reportdarwin-arm64: 70.1 MB
linux-x64: 71.8 MB
win32-x64: 72.7 MB
|
3 participants
These changes significantly improve performance when:
Overview
Problem
Two independent performance/correctness issues:
QueryResultPanel.transmitDataWrapperconstructed each row using object-spread inside aforEach, allocating a new row object per column. For an n-row × m-column result set this is O(n·m²) in allocations and dominates CPU time on wide or large result sets.ConversationProvider.dispose()did not clear the pendingsetTimeoutit used for conversation polling. A scheduled callback could still fire after disposal, keeping the provider alive and executing work against disposed state.Solution
Replace the
forEach+ spread with a single-pass plain for-loop that builds the row into a pre-allocated object, then assigns it once into a pre-sizedrowsarray. Same observable output, no intermediate allocations.Clear the timer at the top of
dispose()and null it out before running the usual disposables cleanup.How to test
Notes
This is a rebase of the earlier version of this PR on current
master. The sync→async file-I/O changes from the previous revision have been dropped:dbtCoreIntegration.tswas extracted into the@altimateai/dbt-integrationpackage (PR refactor: use dbt integration library #1697), so the hunk no longer applies here.dbtTestService.ts,docsEditPanel.ts,newDocsGenPanel.ts) were left out to keep this PR focused on the two clear wins; they can be revisited separately if needed.Checklist
README.mdupdated and added information about my change