Hi, and thank you for maintaining scoringRules.
For context, scoringRules underpins ordinal probability scoring in the hubverse forecast ecosystem (we rely on it indirectly, via scoringutils). The strict sum-to-one check inside rps_probs() can be difficult to satisfy in practice for forecasts in our setting, and the same point has been raised independently at epiforecasts/scoringutils#996, which prompted us to raise it upstream here as well.
Problem
check_p0() (R/rps.R) currently rejects any probability vector whose sum deviates from 1 by more than .Machine$double.eps (~2.2e-16). This is many orders of magnitude tighter than the standard R tolerance for floating-point equality used by all.equal() (sqrt(.Machine$double.eps), ~1.5e-8), which is also the tolerance we use to validate that hubverse forecast probabilities sum to 1. Forecasts that pass our validation can therefore still fail this check.
Current downstream effect
scoringutils recently relaxed its own equivalent check in epiforecasts/scoringutils#997, so it now accepts these forecasts and forwards them to scoringRules::rps_probs(). rps_probs() still rejects them; scoringutils catches the error and demotes it to a Computation for 'rps' failed warning, silently dropping the rps column from the scoring output. Users requesting RPS for an ordinal forecast on realistic data therefore get a warning and a missing metric rather than a score.
Suggestion
Would you consider adding an optional tol argument to rps_probs() (forwarded to check_p0()), defaulting to the current .Machine$double.eps?
This mirrors R's own all.equal(target, current, tolerance = ...) pattern, preserves existing behaviour for current users (the default is unchanged), and gives downstream packages a clean way to opt into a looser tolerance where appropriate.
A blanket loosening of the hard-coded tolerance would also resolve our issue, but would change behaviour for existing callers, which seems less attractive.
We'd be happy to put together a PR if this is welcome.
Thank you for considering, and again for the package.
Hi, and thank you for maintaining scoringRules.
For context, scoringRules underpins ordinal probability scoring in the hubverse forecast ecosystem (we rely on it indirectly, via scoringutils). The strict sum-to-one check inside
rps_probs()can be difficult to satisfy in practice for forecasts in our setting, and the same point has been raised independently at epiforecasts/scoringutils#996, which prompted us to raise it upstream here as well.Problem
check_p0()(R/rps.R) currently rejects any probability vector whose sum deviates from 1 by more than.Machine$double.eps(~2.2e-16). This is many orders of magnitude tighter than the standard R tolerance for floating-point equality used byall.equal()(sqrt(.Machine$double.eps), ~1.5e-8), which is also the tolerance we use to validate that hubverse forecast probabilities sum to 1. Forecasts that pass our validation can therefore still fail this check.Current downstream effect
scoringutils recently relaxed its own equivalent check in epiforecasts/scoringutils#997, so it now accepts these forecasts and forwards them to
scoringRules::rps_probs().rps_probs()still rejects them; scoringutils catches the error and demotes it to aComputation for 'rps' failedwarning, silently dropping therpscolumn from the scoring output. Users requesting RPS for an ordinal forecast on realistic data therefore get a warning and a missing metric rather than a score.Suggestion
Would you consider adding an optional
tolargument torps_probs()(forwarded tocheck_p0()), defaulting to the current.Machine$double.eps?This mirrors R's own
all.equal(target, current, tolerance = ...)pattern, preserves existing behaviour for current users (the default is unchanged), and gives downstream packages a clean way to opt into a looser tolerance where appropriate.A blanket loosening of the hard-coded tolerance would also resolve our issue, but would change behaviour for existing callers, which seems less attractive.
We'd be happy to put together a PR if this is welcome.
Thank you for considering, and again for the package.