Reduced volume LUTs and updated P3 LUTs#145
Merged
isilber merged 1 commit intoARM-DOE:masterfrom May 21, 2025
Merged
Conversation
This commit: 1. Updates all Mie LUTs - file volume reduced by a factor of ~200. This reduction directly speeds up processing by more than 10% and indirectly expedites processing of larger model output files and/or more subcolumns by reducing RAM usage, thereby enabling parallel processing of larger chuncks of data. 2. Updates all P3 LUTs - upon testing error magnitudes stemming from this volume reduction, it became clear that the hydrometeor diameter array used in the P3 LUTs is prone to errors on the order of a few tens of percent, especially at large and very small effective sizes and high mu (narrower distributions). Expansion of the P3 single particle diameter dimension by ~40% to 466 entries (with LUT file total volume increase of 15-20%) reduces these errors to fraction of a percent.
Collaborator
Author
|
@rcjackson, I think this PR is solid, but another pair of eyes on the description and what I've done here before merging would be great. |
Collaborator
|
Looks good to me @isilber ! |
rcjackson
approved these changes
May 21, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR:
Updates all Mie LUTs (including the m-D A-D, or mDAD LUTs used by the
E3SMv1and inherited classes) - file volume reduced by a factor of ~200. This reduction directly speeds up processing by more than 10%. It indirectly expedites the processing of larger model output files and/or more subcolumns by reducing RAM usage, thereby enabling parallel processing of larger chunks of data.Updates all P3 LUTs - upon testing error magnitudes stemming from this volume reduction, it became clear that the hydrometeor diameter array used in the P3 LUTs is prone to errors on the order of a few tens of percent (not shown), especially at large and very small effective sizes and high mu (narrower distributions). Expansion of the P3 single particle diameter dimension by ~40% to 466 entries (with LUT file total volume increase of 15-20%) reduces these errors to fractions of a percent.
The errors associated with the Mie file volume reduction were rigorously examined by evaluating them as a function of in-cloud water content and number concentration. The variable suffering from the most significant errors is the backscatter cross section, with errors generally greater at higher mu (shape parameter) values (most of which are beyond microphysics scheme limits) — see example figure below. Using 144 water content-number concentration combinations for each mu value (out of 16 mu values), and combining all instruments, the analysis results in an average backscatter cross-section error of ~0.2%. The example figure below demonstrates CEIL backscatter cross-section errors, which tend to be larger at the different panels' bottom right corners. However, these enhanced error regions correspond to effective radii < 2 um (see bottom figure), which are largely irrelevant for cloud research. Thus, we conclude that the reduced volume LUTs are robust and that errors are negligible for all practical EMC² purposes.