Sensory: cross-validated discriminator for the observational relate step#433
Merged
Conversation
Two predictive-importance diagnostics next to vip(): target_projection() forms the single latent component along a response's regression vector (Kvalheim and Karstang 1989), and selectivity_ratio() reports each feature's explained-to-residual variance ratio on that component (Rajalahti et al. 2009). Both are bound as PLS convenience methods and exported from multivariate.methods. The selectivity ratio ranks predictive relevance but, by design, gives near-identical scores to two collinear features, so it does not break ties within a collinear group.
Synthetic driver/proxy/noise block: SR is high for the predictive direction (driver and its collinear proxy), low for noise, and gives near-identical scores to the two collinear features (the limitation the discriminator narrative rests on). TP scores correlate with the response. Adds multi-response, error-path, and LDPE real-data checks.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
discriminate_observational adds out-of-sample evidence on top of the marginal associations: a per-attribute cross-validated Q-squared gate (reusing PLS.select_n_components), a selectivity ratio per descriptor on the target-projected predictive direction with a permutation p-value (Benjamini-Hochberg corrected per attribute), and a collinear-cluster id so proxies that ride on a driver are reported as one inseparable group. The permutation loop is skipped for attributes the Q-squared gate finds unpredictable. Surfaced as a 'discriminator' key on the relate output and threaded through analyze_descriptive. Existing fast tests opt out via discriminator=False.
Adds discriminator / n_permutations / random_state options to the sensory_analyze_descriptive tool input and mentions the discriminator output in the tool description. Passes a DataFrame to select_n_components and indexes the selectivity ratio by position so mypy is satisfied.
…minator The per-descriptor permutation now controls multiplicity with a Westfall-Young max-statistic null per attribute, so a single genuine driver is detectable without the resolution loss of a per-test BH floor. The Q-squared gate uses leave-one-out cross-validation (cheap and robust to near-collinear blocks), and PLS fits step the component count down on a singular block. The end-to-end example gains three measurement-condition nuisances; the test now asserts the Q-squared gate, the brix collinear cluster kept as one inseparable group (Trap B), genuine lone drivers kept, and a chance correlate (lab_humidity vs Liking) flagged by the marginal test but demoted by the discriminator (Trap A).
Adds a 'Telling genuine drivers from proxies' section explaining the cross-validated Q-squared gate, the selectivity ratio on the target-projected direction, and collinear clustering, and a Step 5 in the worked example. States plainly that observational cross-validation demotes coincidences and groups proxies with their driver but cannot rank descriptors within a collinear cluster: that needs an external dataset, a designed experiment, or mechanistic knowledge.
Version 1.50.0 (new feature: PLS target projection / selectivity ratio and the cross-validated sensory discriminator), with CITATION and CHANGELOG in sync. Adds unit tests for the diagnostic error paths (response selection, unfitted models, the small-sample F-critical) and for the discriminator's Q-squared gate and collinear clustering.
…torial Add an 'under the hood' note pointing at target_projection and selectivity_ratio with their one-line formulas, plus a References block (Kvalheim and Karstang 1989; Rajalahti et al. 2009), so the worked example is self-contained.
Each @tool_spec description now carries a Returns section enumerating the JSON keys it produces, so an importing agent knows the output contract without inspecting a response first. The analyze tool spells out the full discriminator structure (per_attribute, descriptors, clusters) and notes that descriptors sharing a cluster_id cannot be told apart.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
The observational relate step reported only marginal Pearson correlations
(BH-corrected) between each sensory attribute and each measured descriptor. In
that view a genuine driver and a proxy that rides on it look identical, because
within one dataset they correlate equally, and a chance correlation in a small
product set can pass too. The descriptive-panel tutorial promised that telling
these apart "is covered separately" using out-of-sample evidence. This PR builds
that discriminator.
A limitation is stated plainly throughout, because it drives the design and the
narrative: same-data cross-validation cannot separate two near-collinear
descriptors (one carries the same information as the other, so it predicts just
as well out of sample). What a cross-validated discriminator can do is gate on
whether an attribute is predictable at all, demote correlations that do not
survive cross-validation, and report collinear descriptors as one inseparable
cluster rather than ranking within it. Separating a genuine correlate from a
collinear proxy needs an external dataset, a designed experiment, or mechanistic
knowledge.
What changed
multivariate/_diagnostics.py, bound as PLSconvenience methods and exported from
multivariate.methods:target_projection(): the single latent component along a response'sregression vector (Kvalheim and Karstang, 1989).
selectivity_ratio(): per-feature explained-to-residual variance ratio onthat component (Rajalahti et al., 2009).
discriminate_observational, surfaced asresult.relate["discriminator"]and through thesensory_analyze_descriptivetool): a per-attribute leave-one-out Q-squared gate, a selectivity ratio per
descriptor with a max-statistic (Westfall-Young) permutation test, and
collinear-cluster grouping. PLS fits step the component count down on a
near-collinear (singular) block. On by default; existing fast tests opt out
with
discriminator=False.direct discriminator test, and an extended end-to-end example that asserts the
Q-squared gate, the brix collinear cluster kept as one inseparable group
(Trap B), genuine lone drivers kept, and a chance correlate
(
lab_humidityvs Liking) flagged by the marginal test but demoted by thediscriminator (Trap A).
the worked example, reframed around what observational cross-validation can and
cannot do.
CITATION.cffandCHANGELOG.mdin sync.Verification
ruff check .andmypy src/process_improveclean. Full multivariate suite(167 passed, 1 skipped) and sensory suites (40 passed) green locally.