Data:HELM instances

[borgr]

HELM has more than the aggregated data (and it also has a lot of dataset except classic).

[borgr]

@damian1996 We mentioned it today on the call, what's the status there? Do we want help from others? Need to do it just less piorities than the backbone architecture, something else?

[damian1996]

There is a script that extracts data from 5 leaderboards (Instruct, Classic, MMLU, Capabilities, Lite). I added it before we standardized instance-level information in its current form.

There is also a HELM converter that transforms aggregated and instance-level data into the EEE format for new HELM evals.

So it should be quite fast to extract instance-level info for evaluations from the leaderboards by combining these two scripts. I could try to do it over the next few days (probably next week).

[borgr]

Sounds good, if there are other priorities, data comes last (as few others can contribute to the main code and more others can contribute data). But if not, I am making a bit of order in all of the datasources coming in.

[damian1996]

I'm wondering what is best place to get instance level data for HELM evals. I've found list of runs (example run). But there is missing some info like

[borgr]

@yifanmai are those just really missing sometimes? Or is it the wrong place to look at?

[yifanmai]

The GPQA questions and answers are not stored in plain text in HELM due to this clause in the dataset card for GPQA:

We request that you do not reveal examples from this dataset in plain text or images online, to reduce the risk of leakage into foundation model training corpora.

It might be a good idea to ask the GPQA authors if they think this should still be in effect in 2026. If so, we should avoid putting the text in Every Eval Ever as well.

As for HELM, we comply by using a simple encryption mechanism over the raw data. You can decrypt the using the decryption script like so:

export $SUITE = v1.15.0

# Download script
curl https://raw.githubusercontent.com/stanford-crfm/helm/refs/tags/v0.5.14/scripts/decrypt_scenario_states.py > decrypt_scenario_states.py

# Download data
mkdir -p benchmark_output/runs/$SUITE/gpqa:subset=gpqa_main,use_chain_of_thought=true,use_few_shot=false,model=amazon_nova-lite-v1:0/
curl https://storage.googleapis.com/crfm-helm-public/capabilities/benchmark_output/runs/v1.0.0/gpqa%3Asubset%3Dgpqa_main%2Cuse_chain_of_thought%3Dtrue%2Cuse_few_shot%3Dfalse%2Cmodel%3Damazon_nova-lite-v1%3A0/scenario_state.json > benchmark_output/runs/$SUITE/gpqa\:subset\=gpqa_main\,use_chain_of_thought\=true\,use_few_shot\=false\,model\=amazon_nova-lite-v1\:0/scenario_state.json
curl https://storage.googleapis.com/crfm-helm-public/capabilities/benchmark_output/runs/$SUITE/gpqa%3Asubset%3Dgpqa_main%2Cuse_chain_of_thought%3Dtrue%2Cuse_few_shot%3Dfalse%2Cmodel%3Damazon_nova-lite-v1%3A0/encryption_data.json > benchmark_output/runs/$SUITE/gpqa\:subset\=gpqa_main\,use_chain_of_thought\=true\,use_few_shot\=false\,model\=amazon_nova-lite-v1\:0/encryption_data.json
  
# Decrypt
# Note: Needs crfm-helm installed in Python virtual environment
python3 decrypt_scenario_states.py --suite $SUITE

# After this, scenario_state.json will contain the decrypted text.