Skip to content

Bugfix/257 speed up simpathsmodelpopulatetaxdbreferences process#268

Merged
andrewbaxter439 merged 7 commits intodevelopfrom
bugfix/257-speed-up-simpathsmodelpopulatetaxdbreferences-process
Oct 14, 2025
Merged

Bugfix/257 speed up simpathsmodelpopulatetaxdbreferences process#268
andrewbaxter439 merged 7 commits intodevelopfrom
bugfix/257-speed-up-simpathsmodelpopulatetaxdbreferences-process

Conversation

@andrewbaxter439
Copy link
Copy Markdown
Collaborator

What

  • changes donortaxunit lookups at the start of each new run to lazy join policies then sort in memory

Why

  • this becomes a very slow process for multiple years of tax benefit policies. Should speed up from ~5 mins to ~2 mins in post-setup pre-run phase

@andrewbaxter439 andrewbaxter439 force-pushed the bugfix/257-speed-up-simpathsmodelpopulatetaxdbreferences-process branch from fa6ba3e to 840af45 Compare October 13, 2025 16:04
@andrewbaxter439 andrewbaxter439 force-pushed the bugfix/257-speed-up-simpathsmodelpopulatetaxdbreferences-process branch from 840af45 to 7878c81 Compare October 13, 2025 17:44
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR improves the performance of the SimplPathsModelPopulateTaxdbReferences process by optimizing how donor tax unit lookups are performed at the start of each run. The changes convert eager fetching to lazy loading and perform sorting in memory instead of in the database.

  • Changed from eager to lazy fetching of policies in DonorTaxUnitPolicy
  • Modified query to remove LEFT JOIN FETCH and sort in memory by max year policy income
  • Added database indexes and batch size optimization for better performance

Reviewed Changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 2 comments.

File Description
src/main/java/simpaths/model/taxes/DonorTaxUnitPolicy.java Changed fetch strategy from EAGER to LAZY, added BatchSize annotation and database indexes for performance
src/main/java/simpaths/model/SimPathsModel.java Modified donor pool query to remove JOIN FETCH and implemented in-memory sorting by max year policy income
src/test/java/simpaths/integrationtest/expected/Statistics21.csv Updated expected test results with minor numerical differences due to performance optimization changes
src/test/java/simpaths/integrationtest/expected/HealthStatistics1.csv Updated expected test results with minor numerical differences due to performance optimization changes

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@andrewbaxter439
Copy link
Copy Markdown
Collaborator Author

To sense-check: this produces miniscule differences in calculated stats, usually beyond the 7th/8th decimal places in only about a 10th of estimates. The only way I think this is possible is due to differing de-duplication procedures in the searching approaches?

@andrewbaxter439 andrewbaxter439 force-pushed the bugfix/257-speed-up-simpathsmodelpopulatetaxdbreferences-process branch from b5c2207 to 73ea1d5 Compare October 14, 2025 11:59
@andrewbaxter439 andrewbaxter439 merged commit 6d10ff0 into develop Oct 14, 2025
5 checks passed
@andrewbaxter439 andrewbaxter439 deleted the bugfix/257-speed-up-simpathsmodelpopulatetaxdbreferences-process branch January 6, 2026 10:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants