[FEATURE] Migrate statistical distribution expectations to the V1 API#11962
Draft
joshua-stauffer wants to merge 5 commits into
Draft
[FEATURE] Migrate statistical distribution expectations to the V1 API#11962joshua-stauffer wants to merge 5 commits into
joshua-stauffer wants to merge 5 commits into
Conversation
…ions API Implement and export four previously-stubbed statistical Expectations that raised NotImplementedError and were excluded from the public API: - ExpectColumnChisquareTestPValueToBeGreaterThan - ExpectColumnBootstrappedKsTestPValueToBeGreaterThan - ExpectColumnParameterizedDistributionKsTestPValueToBeGreaterThan - ExpectColumnPairCramersPhiValueToBeLessThan Chi-square computes its statistic in _validate from column value counts and non-null count; the KS Expectations wire up their existing Pandas metric providers (registering the bootstrapped-KS provider and fixing its value_keys); Cramér's phi is a batch-level association measure over two columns computed from the full frame. Also fixes the parameterized-distribution KS metric to freeze the requested distribution and test against its CDF, which is required by current SciPy. Each Expectation is Pandas-only, matching the backends its underlying computation supports. Adds integration tests under data_sources_and_expectations and generated JSON schemas for each new Expectation.
✅ Deploy Preview for niobium-lead-7998 canceled.
|
…-test timeout The default 1000 bootstrap rounds pushed each validation to ~1.9s, exceeding the 2s per-test timeout in CI. Use 100 rounds, which keeps each test well under the limit while the matching/skewed partitions still give a stable pass/fail.
Double-indexing to_json_dict()["result"]["observed_value"] is not type-safe (the value is a union that includes list). Compare the whole result dict with pytest.approx inside, matching the pattern used by the other expectation tests.
- Fix inverted precision in numeric interval binning: small bin widths (< ~0.01) rounded to duplicate interval labels and crashed Categorical.from_codes. Precision now grows as bin width shrinks. - Detect all numeric dtypes (int32/float32, nullable Int64/Float64) via is_numeric_dtype instead of a hardcoded list that silently routed them to the categorical branch; keep booleans on the categorical branch. - Introduce the "(missing)"/"(other)" sentinels without raising on already-categorical columns (cast to object first). - Forward the bootstrapped-KS metric details verbatim so observed/expected partitions and CDFs reach the result. - Document the inclusive boundary comparisons (<= / >=) that mirror legacy behavior for Cramér's phi and parameterized-distribution KS. - Fix the self-contradictory Cramér's phi failing docstring example to use perfectly-associated data. - Add integration tests: numeric binning (incl. small-magnitude), explicit bins/n_bins, nulls, category dtype, and chi-square tail_weight_holdout.
Chained subscripts into to_json_dict()["result"] return JSONValues (a union) and fail mypy's __getitem__ overload check. Access result.result (a plain dict) behind an is-not-None guard instead, matching the pattern used elsewhere in the integration tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four statistical distribution Expectations existed only as stubs that raised
NotImplementedErrorand were excluded from the public API (tagged "needsmigration to modular expectations api"). This PR completes their migration to the
modular Expectations API and exports them so they are usable and registered:
ExpectColumnChisquareTestPValueToBeGreaterThanExpectColumnBootstrappedKsTestPValueToBeGreaterThanExpectColumnParameterizedDistributionKsTestPValueToBeGreaterThanExpectColumnPairCramersPhiValueToBeLessThanThese are the same Expectations that #11764 proposed removing as dead code; they
are dead only because they were never migrated. Rather than delete them, this PR
implements them.
What changed
ExpectColumnChisquareTestPValueToBeGreaterThan): computes thegoodness-of-fit p-value in
_validatefromcolumn.value_countsandcolumn_values.nonnull.count(same approach as the categorical branch ofExpectColumnKLDivergenceToBeLessThan). Success when the p-value exceedsp.(
ExpectColumnParameterizedDistributionKsTestPValueToBeGreaterThan): wires upthe existing
column.parameterized_distribution_ks_test_p_valuemetric. Successwhen the KS p-value is at least
p_value.ExpectColumnBootstrappedKsTestPValueToBeGreaterThan):registers the existing (but previously unimported)
column.bootstrapped_ks_test_p_valuemetric and fixes its
value_keyssobootstrap_samplesis actually passedthrough. Success when the bootstrapped statistic exceeds
p.ExpectColumnPairCramersPhiValueToBeLessThan): a batch-levelmeasure of association between two columns. Builds the contingency table
(binning numeric columns) and computes Cramér's V from the Chi-square statistic.
Success when V is at most
threshold.scipy.stats.kstest(column, distribution, args=...), which current SciPy nolonger supports for parameterless CDFs (e.g.
norm). It now freezes therequested distribution and tests against its
.cdf, which is equivalent andworks across SciPy versions.
Backends
The underlying computations (SciPy KS tests, bootstrapping, contingency-table
Cramér's V) are implemented for Pandas only, matching where these statistics were
historically available. Each Expectation declares Pandas as its only supported
data source. Extending to SQL/Spark would be follow-up work.
Tests & artifacts
tests/integration/data_sources_and_expectations/expectations/for each Expectation (success, failure, and suite-parameter cases), following the
existing per-Expectation Pandas patterns. Because the bootstrapped-KS statistic is
randomized, its tests assert on success/failure rather than the exact value.
invoke schemas --sync) and addedthem to the supported-expectations list.
Verification performed
-m unit).test_core_model_schemas.pypasses (serializable, schemas updated, valid spec).ruff check,ruff format).third-party stubs; CI mypy is authoritative).