[feat] Add model stability measurement (EA, CA, IV, CR) for k-samples mode #997
+172
−3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add four new metrics for measuring model stability in k-samples mode: - expected_accuracy: mean accuracy across all k samples - consensus_accuracy: accuracy after majority voting - internal_variance: average variance within each question (lower is better) - consistency_rate: fraction of questions with consistent answers Reference: HackMD v0.6 roadmap section 2.5 Model Stability Measurement
- Add calculate_stability_metrics() method to TaskOutput class Groups scores by question and computes EA, CA, IV, CR when repeats > 1 - Update consolidate_results() to include stability metrics in output The metrics are only computed when num_samples > 1 (k-samples mode).
- Override task repeats with num_samples when n > 1 for stability measurement - Call calculate_stability_metrics() after aggregate metric calculation When --num_samples is set > 1, the evaluator runs each question k times to measure model consistency and stability.
…ment Add CLI argument to enable k-samples mode: -n, --num_samples: Number of samples per question (default: 1) When n > 1, enables k-samples mode and computes stability metrics (EA, CA, IV, CR) to measure model consistency. Usage: lmms-eval --model xxx --tasks xxx -n 5
…able - Add EA, CA, IV, CR columns to make_table() output - Skip stability metric variants in main metric loop (shown as columns) Example output: |Task|Metric|Value|Stderr|Stderr_CLT|Stderr_Clustered|EA |CA |IV |CR | |----|------|-----|------|----------|----------------|----|----|----|----| |mme |score |85.0 |N/A |0.0435 |0.0512 |0.80|0.82|0.05|0.75|
mwxely
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds model stability measurement capabilities to
lmms-eval, enabling users to assess model consistency by running multiple samples per question.Motivation
Same accuracy does not mean same reliability:
Model A and B have the same Expected Accuracy, but Model A is 3× more stable (lower IV).
Changes
New CLI Parameter
lmms-eval --model xxx --tasks xxx -n 5 # or --num_samples 5When
n > 1, enables k-samples mode and computes stability metrics.New Metrics
Example Output
Files Changed
lmms_eval/api/metrics.pyexpected_accuracy,consensus_accuracy,internal_variance,consistency_ratefunctionslmms_eval/evaluator_utils.pycalculate_stability_metrics()methodlmms_eval/evaluator.pynum_samples > 1, call stability calculationlmms_eval/__main__.py-n/--num_samplesCLI parameterlmms_eval/utils.pyTest Results
--num_samples 3on VideoMMEcicdtest