Project Overview: Update TMD national and area data

[donboyd5]

Goals

• Update national TMD data to use 2022 IRS data and 2022 CPS, and area data to use 2022 IRS data
• Update outyears to reflect recent economic history and current economic forecasts
• Produce updated area weights for all states and all Congressional Districts

This is the master issue - showing the anticipated overall project structure and progress. Thanks to @martinholmer for advice on how to do this.

The next comment shows tasks and subtasks. I will update it periodically as I complete tasks and make adjustments to project organization.

[donboyd5]

Update April 2026 — project closed. This task list reflects the project as originally scoped and was not kept current as the scope expanded and work progressed. For the complete picture — including work beyond the original scope (CD-119 crosswalk, full R→Python pipeline replacement, quality reports, developer tools, documentation overhaul) and a mapping of accomplishments to PRs — see the project closeout comment below.

Tasks and subtasks

In addition to the changes below, the project has benefited greatly from numerous process, infrastructure, and code changes, and fixes by @martinholmer, including: #417, #428, #431, #432, #434, #435, #437, #438, #439, #440, #441, #443, #444, #445, #447, #451, #453, #454, #461, #467, #468, #469.

TMD infrastructure enhancements and fixes

• Fix SSL certificate verification error preventing CPS data downloads #354 -- addressed by Fix SSL certificate verification for CPS downloads -- cleaner version #357
• Fix pandas ChainedAssignmentError warning in create_taxcalc_growth_factors.py (not essential in short term) #373-- addressed by PR fix pandas chained assignment #374
• Tax expenditure tests fail when using Tax-Calculator 5.2.0 but not 5.1.0 #356 -- addressed by Fix failing taxcalc 5.2.0 tests #362
• Add GPU acceleration of national reweighting when GPU is available, falling back to CPU otherwise #366 -- addressed by Add GPU Acceleration for National Weight Optimization #367
• Ignore local venv -- addressed by ignore local venv #361
• Add claude/ to .gitignore
  -- addressed by Add claude/ to .gitignore #359
• Add .quarto/ directories to .gitignore
  -- addresssed by Add .quarto/ directories to .gitignore #358
• Reduce cross-machine differences in record weights and improve national weighting performance #400 -- addressed by Improve reweighting: higher penalty, more iterations, gradient convergence #407. Further improvements may be obtained by using the approach described in Improve cross-machine reproducibility, quality, and speed of national weight optimization by reformulating as QP and switching to scipy L-BFGS-B #413. Significant improvements made in Update TMD version from 1.1.0 to 1.2.0 #417, which addressed Change national reweighting objective function and replace PyTorch L-BFGS with Clarabel QP solver #416.
• Address deprecation warning in scipy minimize call to L-BFGS-B (low priority) -- Initially addressed by Address #360: Replace deprecated iprint in scipy L-BFGS-B for area weights with a callback function #418. New pipeline uses Clarabel so no longer needed.
• Create reproducible environments for subprojects in R #379 Closed. Unnecessary because new target creation pipelines are in python. Old R infrastructure removed with Remove old R/Quarto national targets pipeline; will be replaced with a much-better python pipeline #449, Prepare for a python state target creation pipeline by removing old R/Quarto state target preparation pipeline #462, Remove old R/Quarto CD pipeline, weight examination, and stale recipes #463.
• Miscellaneous. Address #419: Fix ambiguous AGI range labels in reweighting output #421, Fix #420, tolerances for tax expenditure tests #422, Change "e00100" to "c00100" to properly loosen AGI area weights test … #436, Address #455: Fix double-scaling of E00900 and E26270 during PUF uprating #456, Restore xx_targets.csv and xx_params.yaml test fixtures #464.
• Address skipped tests. Improve cross-machine reproducibility "fingerprint" test for area weights #477.

---

Update national tmd data to 2022 base year and grow from there

• Prepare initial folder structure for national target development and reproduce 2021 national targets #370 -- addressed by Add potential 2022 targets data to national_targets folder #376, Add python IRS-to-national-targets pipeline #450,
• Ensure that newly-included potential national targets for 2022 are the values that were intended #377 -- Completed in the course of preparing PR Add 2022 IRS target data infrastructure for national reweighting #424
• Map and document relationships between PUF variables and IRS spreadsheets -- Completed in the course of preparing PR Add 2022 IRS target data infrastructure for national reweighting #424
• Closed. Already addressed by @martinholmer in Add create_tc_cps(taxyear) function to the tmd/datasets/cps.py module #432. Update CPS data to the 2022 income year #369 -- to come
• Update TMD targets and targeting process to reflect the 2022 targets -- addressed by PR Add 2022 IRS target data infrastructure for national reweighting #424, Parameterize TAXYEAR to ease transition from a 2021-targeted national tmd file to a 2022-targeted file #425, and Add python IRS-to-national-targets pipeline #450
• Develop additional comparisons and tests for the new data -- Addressed by Add python IRS-to-national-targets pipeline #450.
• Update growfactors as appropriate. Population updating addressed by @martinholmer in Update population projection and growfactors for TAXYEAR=2022 #447.

---

Construct weights for states based on 2022 IRS SOI Historical Table 2 data for states

• Obtain and computerize 2022 IRS state targeting data from SOI Historical Table 2, Add Python state target preparation pipeline #465
• Updating and, as appropriate, extend mapping of IRS reporting concepts to PUF concepts (i.e., which IRS table-column-row or combination of these corresponds most closely to a particular PUF variable or combination of PUF variables?)
• Develop proxy linkages as needed, where an SOI HT2 variable, or combination of variables, is used to apportion a related PUF variable, or combination of variables, across states Use Census State and Local Finance data to construct SALT available itemized deduction growth rate for 2015-2022 #460
• Incorporate updated data into the TMD framework Solve for state weights #466, Improve area weight solver: robustness, memory, and testing #470
• Improve documentation
• Analyze results
• Tests

---

Construct weights for Congressional Districts based on 2022 IRS SOI historical data

• Obtain and computerize 2022 IRS CD targeting data from SOI historical tables
• Update and, as appropriate, extend mapping of IRS reporting concepts to PUF concepts (i.e., which IRS table-column-row or combination of these corresponds most closely to a particular PUF variable or combination of PUF variables?) Add spec-based target pipeline with SOI CD data ingestion #471
• Develop proxy linkages as needed, where an SOI historical data variable, or combination of variables, is used to apportion a related PUF variable, or combination of variables, across CDs
• Incorporate updated data into the TMD framework
• Improve documentation
• Analyze results
• Tests

[donboyd5]

Background

The current base year for tax microdata benchmarking (tmd) is 2021. The national file is targeted in detail to 2021 levels, and grows from there (with growfactors). State and Congressional District weights were also developed based on 2021 data. The reason for this was data availability: At the time the tmd data were prepared, the greatest amount of detailed data were available for 2021.

Now, detailed IRS targeting data are available nationally, for states, and for Congressional Districts for 2022. This, in concept, is a much better year to use for targeting because 2021 was severely affected by disruptions related to COVID and to economic and policy responses to COVID. 2022 is likely to be far less affected by COVID-related disruptions.

Under an agreement with PSL, I will update the national data to 2022, and growth factors to later years, and then update state and Congressional District weights using these updated national data, allocated to states and CDs relying heavily on the now-available 2022 IRS data for states and CDs.

[donboyd5]

Project closeout — April 2026

This umbrella issue is now complete. A summary report covers project goals, the most consequential accomplishments, work that exceeded the original scope, complementary contributions by @martinholmer, and suggested next steps.

This comment maps the project's accomplishments to the PRs that delivered them, organized by the same eight headings used in the report. The earlier task-list comment is left in place as historical record of the project as originally scoped.

PR-author attributions in this comment were verified individually
against gh api .../pulls/<n> before posting.

1. Higher-quality, reproducible weighting (Clarabel QP)

• Improve reweighting: higher penalty, more iterations, gradient convergence #407 — improve reweighting: higher penalty, more iterations, gradient convergence
• Change national reweighting objective function and replace PyTorch L-BFGS with Clarabel QP solver #416 — replace PyTorch L-BFGS with Clarabel QP solver and redesign the objective function (the centerpiece of the reweighting overhaul)
• Improve area weight solver: robustness, memory, and testing #470 — improve area-weight solver robustness, memory, and testing
• Address #360: Replace deprecated iprint in scipy L-BFGS-B for area weights with a callback function #418 — replace deprecated scipy iprint with a callback (initially aimed at area weights; superseded by Clarabel for the national pipeline)
• Streamline national reweighting algorithm #437 (@martinholmer) — streamline national reweighting algorithm (related cleanup)

2. National targets and microdata refreshed to 2022

• Add 2022 IRS target data infrastructure for national reweighting #424 — 2022 IRS target data infrastructure for national reweighting
• Parameterize TAXYEAR to ease transition from a 2021-targeted national tmd file to a 2022-targeted file #425 — parameterize TAXYEAR
• Add python IRS-to-national-targets pipeline #450 — Python IRS-to-national-targets pipeline
• Use Census State and Local Finance data to construct SALT available itemized deduction growth rate for 2015-2022 #460 — SALT itemized-deduction growth rate from Census State and Local Finance data
• Add create_tc_puf(taxyear) function to the tmd/datasets/puf.py module #431 (@martinholmer) — create_tc_puf(taxyear)
• Add create_tc_cps(taxyear) function to the tmd/datasets/cps.py module #432 (@martinholmer) — create_tc_cps(taxyear)
• Calibrate CPS filer parameters so population target is hit without scaling CPS nonfiler weights #438 (@martinholmer) — calibrate CPS filer parameters
• Fix uprate_puf.py logic to avoid crash when TAXYEAR=2022 #439 (@martinholmer) — fix uprate_puf.py for TAXYEAR=2022
• Calibrate several assumption parameters for TAXYEAR=2022 #444 (@martinholmer) — calibrate assumption parameters for TAXYEAR=2022
• Update population projection and growfactors for TAXYEAR=2022 #447 (@martinholmer) — population projection and growfactors for TAXYEAR=2022
• Recalibrate imputation parameters after changes in PRs 456 and 460 #461 (@martinholmer) — recalibrate imputation parameters
• Copy final version of growfactors.csv from Tax-Calculator after T-C PR 3006 merged #475 (@martinholmer) — final growfactors.csv

3. State and CD targets refreshed to 2022

• Add Python state target preparation pipeline #465 — Python state target preparation pipeline
• Solve for state weights #466 — solve for state weights
• Add spec-based target pipeline with SOI CD data ingestion #471 — spec-based target pipeline with SOI CD data ingestion
• Preparing for CD weighs pipeline: Add per-bin CTC targets to state spec (169 → 175 targets), to match CD treatment of CTC #473 — per-bin CTC targets in state spec (matching CD treatment)
• Add Congressional District weight solving pipeline #474 — Congressional District weight solving pipeline
• Add 2022 SOI congressional district documentation guide #486 — 2022 SOI Congressional District documentation guide

4. Congress-119 boundary crosswalk

• Add 119th Congressional Session support to CD pipeline (GH #478) #479 — 119th Congressional Session support in the CD pipeline

5. End-to-end Python pipelines (R retired)

• Remove old R/Quarto national targets pipeline; will be replaced with a much-better python pipeline #449 — remove old R/Quarto national targets pipeline
• Add python IRS-to-national-targets pipeline #450 — Python IRS-to-national-targets pipeline (replacement)
• Prepare for a python state target creation pipeline by removing old R/Quarto state target preparation pipeline #462 — remove old R/Quarto state target preparation pipeline
• Remove old R/Quarto CD pipeline, weight examination, and stale recipes #463 — remove old R/Quarto CD pipeline and stale recipes

6. Quality reports and developer tools

• Enhance quality report with CD support and diagnostics #472 — quality report with CD support and diagnostics
• Redesign the area-weight reproducibility fingerprint (closes #477) #484 — area-weight reproducibility fingerprint redesign (closes Improve cross-machine reproducibility "fingerprint" test for area weights #477)
• Add Makefile for national targets pipeline (closes #491) #494 — Makefile for the national targets pipeline (closes Add a Makefile for the national targets pipeline (parallel to tmd/areas/Makefile) #491)
• Add unweighted national-file fingerprint test (closes #503) #504 — unweighted national-file fingerprint test (closes Add an unweighted national-file fingerprint test, paralleling the existing area-weights fingerprint #503)
• Add 2022 SOI sanity-check test, retire superseded variable totals tests (closes most of #501) #508 — 2022 SOI sanity-check test (also under §7)
• Add average AGI, unusualness, and Kish ESS to quality report and write a per-area CSV for every area on every run #509 — quality report enhancements: average AGI, unusualness, Kish ESS, per-area CSV
• Restore real-time terminal output during pipeline stages (fixes regression introduced in PR 497) #510 — restore real-time terminal output during pipeline stages

7. Materially expanded testing

• Increase several test tolerances to ensure tests pass on more computers #411 (@martinholmer) — loosen test tolerances for cross-machine reliability
• Generalize tests so that expected test results can vary by TAXYEAR #440 (@martinholmer) — generalize tests so expected results can vary by TAXYEAR
• Restore xx_targets.csv and xx_params.yaml test fixtures #464 — restore xx_targets.csv and xx_params.yaml test fixtures
• Add tests/__init__.py file #468 (@martinholmer) — add tests/__init__.py
• Exclude area-pipeline tests from root make test #493 — exclude area-pipeline tests from root make test
• Delete skipped test_create_file (partially addresses #430 and #495) #496 — delete skipped test_create_file
• Capture pipeline warnings and surface test-run warnings (addresses #430, #495) #497 — capture pipeline warnings and surface them in test runs
• Remove tests superseded by #504 and orphan 2021 expected-revenue files (part of #430 / #501 cleanup) #507 — remove tests superseded by Add unweighted national-file fingerprint test (closes #503) #504 and orphan 2021 expected-revenue files
• Add 2022 SOI sanity-check test, retire superseded variable totals tests (closes most of #501) #508 — 2022 SOI sanity-check test

8. Substantially improved documentation

• Add 2022 SOI congressional district documentation guide #486 — 2022 SOI CD documentation guide (also under §3)
• Document SOI reference PDFs in national_targets/docs #500 — document SOI reference PDFs in national_targets/docs
• Improve area targeting documentation for users #512 — improve area targeting documentation for users
• Improve developer-facing area weights documentation (issue #506) #513 — improve developer-facing area weights documentation (closes Improve area weights documentation #506)

Plus the in-repo tmd/areas/AREA_WEIGHTING_LESSONS.md and the
restructured READMEs in tmd/areas/, tmd/areas/weights/, and
tmd/national_targets/.

Beyond the original scope

• All 436 Congressional Districts weighted (the original scope
  envisioned CDs but with less comprehensive review than states; the
  CDs ended up receiving the same target-quality standards).
• Congress-119 boundary crosswalk (Add 119th Congressional Session support to CD pipeline (GH #478) #479) on top of the Congress-118
  crosswalk required by the original scope.
• R-based area-targeting code replaced (not just supplemented) by a
  Python pipeline (Remove old R/Quarto national targets pipeline; will be replaced with a much-better python pipeline #449, Add python IRS-to-national-targets pipeline #450, Prepare for a python state target creation pipeline by removing old R/Quarto state target preparation pipeline #462, Remove old R/Quarto CD pipeline, weight examination, and stale recipes #463).
• Quality-report system, fingerprinting, parameter sweeps, batch
  tooling.
• AREA_WEIGHTING_LESSONS.md as a forward-looking maintenance aid.
• Many review-identified issues closed in the course of the project
  rather than deferred (e.g., Fix SSL certificate verification error preventing CPS data downloads #354, Tax expenditure tests fail when using Tax-Calculator 5.2.0 but not 5.1.0 #356, Add GPU acceleration of national reweighting when GPU is available, falling back to CPU otherwise #366, Fix pandas ChainedAssignmentError warning in create_taxcalc_growth_factors.py (not essential in short term) #373, Reduce cross-machine differences in record weights and improve national weighting performance #400, Handle the six skipped tests #430,
  Improve cross-machine reproducibility "fingerprint" test for area weights #477, openpyxl is an undeclared runtime dependency #480, Add a Makefile for the national targets pipeline (parallel to tmd/areas/Makefile) #491, Preserve pipeline warnings after retiring test_create_file (part of #430) #495, Plan for handling the remaining five skipped tests (follow-on to #430) #501, Add an unweighted national-file fingerprint test, paralleling the existing area-weights fingerprint #503, Improve area weights documentation #506).

Complementary contributions by @martinholmer

Grouped by theme. All PR numbers verified.

• CPS and PUF construction modernization: Add create_tc_puf(taxyear) function to the tmd/datasets/puf.py module #431, Add create_tc_cps(taxyear) function to the tmd/datasets/cps.py module #432, Calibrate CPS filer parameters so population target is hit without scaling CPS nonfiler weights #438.
• Tax-year parameterization: Fix uprate_puf.py logic to avoid crash when TAXYEAR=2022 #439.
• 2022 calibration, growfactor, and projection updates: Calibrate several assumption parameters for TAXYEAR=2022 #444,
  Update population projection and growfactors for TAXYEAR=2022 #447, Recalibrate imputation parameters after changes in PRs 456 and 460 #461, Copy final version of growfactors.csv from Tax-Calculator after T-C PR 3006 merged #475.
• OBBBA deduction imputation: Impute three variables related to new OBBBA deductions #397.
• Reweighting cleanup and test reliability: Streamline national reweighting algorithm #437 (streamline
  national reweighting), Increase several test tolerances to ensure tests pass on more computers #411 (test tolerances).
• Code hygiene, refactors, and obsolete-file removal: Simplify Makefile #428, Move MICE params to imputation_assumptions.py plus other code streamlining #434,
  Streamline create_taxcalc_growth_factors.py and require taxcalc package 6.4.1+ #441, Use TAXYEAR to define LAST_YEAR in the create_taxcalc_sampling_weights.py module #443, Make sure tc_variable_metadata.yaml is up-to-date with Tax-Calculator GitHub repo #445, Cosmetic revision to several comments #451, Remove unneeded tmd/storage/input/imputation_parameters.yaml file #453, Remove obsolete tmd/storage/input/agi_targets.csv file #454, Cosmetic changes to cleanup code #467, Refactor soi_state_data.py code to eliminate pandas PerformanceWarning #498, Remove obsolete TMD/results folder #499.
• Test scaffolding: Generalize tests so that expected test results can vary by TAXYEAR #440, Add tests/__init__.py file #468.
• Bug fixes: Fix issues #487, #488, and #489 #490.
• Version releases: Update TMD version from 1.1.0 to 1.2.0 #417 (1.2.0 mid-project), Update version to 2.0.0 which requires taxcalc 6.5.0+ #476 (2.0.0).

Earlier infrastructure (foundation for the rest)

• Fix SSL certificate verification for CPS downloads issue 354 #355, Fix SSL certificate verification for CPS downloads -- cleaner version #357 — fix SSL certificate verification for CPS downloads
  (issue Fix SSL certificate verification error preventing CPS data downloads #354).
• ignore local venv #361, Add .quarto/ directories to .gitignore #358, Add claude/ to .gitignore #359 — .gitignore updates for venv, .quarto,
  claude/.
• Add GPU Acceleration for National Weight Optimization #367 — GPU acceleration of national reweighting (issue Add GPU acceleration of national reweighting when GPU is available, falling back to CPU otherwise #366).
• Add national_targets folder structure for quarto project #372, Add potential 2022 targets data to national_targets folder #376, Apply national_targets changes from ensure-2022-potential-targets-dat… #382 — initial national-targets folder structure and
  2022 potential targets data.

Closing

Closing this umbrella issue. Suggested follow-on work is listed in the
project report's "Potential improvements going forward" section. The
most prominent items:

1. Bring in additional policy-relevant variables, possibly through statistical
   matching against CPS, ACS, SCF, or other external microdata.
2. Target the future distribution of income, not just future levels.
3. Extend area targeting below the CD level (e.g., major counties).
4. Incorporate 2023 SOI data when released.
5. Strengthen institutional sustainability for ongoing TMD maintenance.
6. Additional developer tools could reduce the experimentation and effort
   required for data updating and target selection.
7. Make policy-counterfactual analysis at state and CD levels a first-class
   workflow.