Skip to content

ENIR .fit returns an error when AUC equals 1. #64

@eugeneberta

Description

@eugeneberta

Hi, thanks for the great package!

It seems netcal.binning.ENIR returns an error when .fit is called with separable binary probabilities (predictions for which AUC = 1).
Here is a minimal example to reproduce the bug:

import numpy as np
from netcal.binning import ENIR
cal = ENIR()
cal.fit(np.array([0,1]), np.array([0,1]))

returns the error message

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[1], [line 4](vscode-notebook-cell:?execution_count=1&line=4)
      2 from netcal.binning import ENIR
      3 cal = ENIR()
----> [4](vscode-notebook-cell:?execution_count=1&line=4) cal.fit(np.array([0,1]), np.array([0,1]))

File /opt/anaconda3/envs/calib/lib/python3.12/site-packages/netcal/Decorator.py:62, in dimensions.<locals>.check_dim.<locals>.new_f(*args, **kwds)
     60     elif type(d) == int:
     61         assert len(a.shape) == d, "dimension of arg \'%s\' must match %s but is %d" % (f.__code__.co_varnames[i], str(d), len(a.shape))
---> [62](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/calib/lib/python3.12/site-packages/netcal/Decorator.py:62) return f(*args, **kwds)

File /opt/anaconda3/envs/calib/lib/python3.12/site-packages/netcal/binning/ENIR.py:270, in ENIR.fit(self, X, y)
    267 model_list.pop()
    269 # get model scores and binning models by elbow method
--> [270](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/calib/lib/python3.12/site-packages/netcal/binning/ENIR.py:270) self._model_scores, self._binning_models = self._elbow(X, y, model_list, self.score_function, alpha=0.001)
    271 return self

File /opt/anaconda3/envs/calib/lib/python3.12/site-packages/netcal/Decorator.py:35, in accepts.<locals>.check_accepts.<locals>.new_f(*args, **kwds)
     33     else:
     34         assert isinstance(a, t), "arg \'%s\' does not match %s" % (f.__code__.co_varnames[i],t)
---> [35](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/calib/lib/python3.12/site-packages/netcal/Decorator.py:35) return f(*args, **kwds)

File /opt/anaconda3/envs/calib/lib/python3.12/site-packages/netcal/AbstractCalibration.py:499, in AbstractCalibration._elbow(self, confidences, ground_truth, model_list, score_function, alpha)
    473 @accepts(np.ndarray, np.ndarray, list, str, float)
    474 def _elbow(self, confidences: np.ndarray, ground_truth: np.ndarray, model_list: List['AbstractCalibration'],
    475            score_function: str = 'BIC', alpha: float = 0.001) -> Tuple[List[float], List['AbstractCalibration']]:
    476     """
    477     Select models by Bayesian score and discard models below a certain threshold with elbow method.
    478 
   (...)
    496         Tuple with two lists of (normed) model scores and list of models.
    497     """
--> [499](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/calib/lib/python3.12/site-packages/netcal/AbstractCalibration.py:499)     model_scores = self._calc_model_scores(confidences, ground_truth, model_list, score_function)
    501     num_models = len(model_scores)
    502     score_variance = np.var(model_scores)

File /opt/anaconda3/envs/calib/lib/python3.12/site-packages/netcal/Decorator.py:35, in accepts.<locals>.check_accepts.<locals>.new_f(*args, **kwds)
     33     else:
     34         assert isinstance(a, t), "arg \'%s\' does not match %s" % (f.__code__.co_varnames[i],t)
---> [35](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/calib/lib/python3.12/site-packages/netcal/Decorator.py:35) return f(*args, **kwds)

File /opt/anaconda3/envs/calib/lib/python3.12/site-packages/netcal/AbstractCalibration.py:469, in AbstractCalibration._calc_model_scores(self, confidences, ground_truth, model_list, score_function)
    466     raise ValueError("Unknown score function \'%s\'. Fix your implementation")
    468 # calculate relative likelihood of each model and normalize scores
--> [469](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/calib/lib/python3.12/site-packages/netcal/AbstractCalibration.py:469) model_scores = np.exp((np.min(score) - score) / 2.)
    471 return model_scores

File /opt/anaconda3/envs/calib/lib/python3.12/site-packages/numpy/core/fromnumeric.py:2953, in min(a, axis, out, keepdims, initial, where)
   2836 @array_function_dispatch(_min_dispatcher)
   2837 def min(a, axis=None, out=None, keepdims=np._NoValue, initial=np._NoValue,
   2838         where=np._NoValue):
   2839     """
   2840     Return the minimum of an array or minimum along an axis.
   2841 
   (...)
   2951     6
   2952     """
-> [2953](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/calib/lib/python3.12/site-packages/numpy/core/fromnumeric.py:2953)     return _wrapreduction(a, np.minimum, 'min', axis, None, out,
   2954                           keepdims=keepdims, initial=initial, where=where)

File /opt/anaconda3/envs/calib/lib/python3.12/site-packages/numpy/core/fromnumeric.py:88, in _wrapreduction(obj, ufunc, method, axis, dtype, out, **kwargs)
     85         else:
     86             return reduction(axis=axis, out=out, **passkwargs)
---> [88](https://file+.vscode-resource.vscode-cdn.net/opt/anaconda3/envs/calib/lib/python3.12/site-packages/numpy/core/fromnumeric.py:88) return ufunc.reduce(obj, axis, dtype, out, **passkwargs)

ValueError: zero-size array to reduction operation minimum which has no identity

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions