Evaluation datasets for OpenMath

[YenChen-Wu]

Hi,

We try to reproduce the results of OpenMath Llama (Table 3 in the paper), but the two eval tasks in the scripts (gsm8k_openmath2, math_500_openmath2) are not available from the lm_eval. Just wondering if you use a specific version of lm_eval or other evaluation tools for these datasets. Thank you!

[bminixhofer]

Hi!

Good catch, sorry for the inconvenience!

This uses a fork of lm-eval at https://github.com/bminixhofer/lm-evaluation-harness.

For example, the gsm8k_openmath2 is here: https://github.com/bminixhofer/lm-evaluation-harness/blob/main/lm_eval/tasks/gsm8k/gsm8k-openmath2.yaml