Difficulty Matching Code Results with the Leaderboard

[ZEXUANW]

I find that my coding results capture more informative (metric) than the leaderboard metric, and I am thinking about how to reconcile them.

Could you please advise on how to match the results from the code to the leaderboard? It seems that each metric has three versions in the code, whereas only one version is reported on the leaderboard.

• dataset_id: op
  date_created: 17-12-2025
  file_size: 23048
  method_id: pearson_corr
  metric_ids:
  • sem_grn
  • sem_hvg
  • sem
    metric_values:
  • '0.0987688622048502'
  • '0.07334421631793887'
  • '0.08605653926139453'
• dataset_id: op
  date_created: 17-12-2025
  file_size: 23048
  method_id: pearson_corr
  metric_ids:
  • tfb_precision
  • tfb_recall
  • tfb_f1
    metric_values:
  • '0.04776565481165778'
  • '0.02756733797997705'
  • '0.03495870537547806'
• dataset_id: op
  date_created: 17-12-2025
  file_size: 23048
  method_id: pearson_corr
  metric_ids:
  • gs_precision
  • gs_recall
  • gs_f1
  • gs_n_active
    metric_values:
  • '0.70174168789988'
  • '0.3279813455378404'
  • '0.40692143797401853'
  • '348.6666666666667'
• dataset_id: op
  date_created: 17-12-2025
  file_size: 23048
  method_id: pearson_corr
  metric_ids:
  • r2_raw
  • r_precision
  • r_recall
  • r_f1
    metric_values:
  • '0.6934894982541678'
  • '0.49783144316145583'
  • '0.273573899861934'
  • '0.35310538256265267'
• dataset_id: op
  date_created: 17-12-2025
  file_size: 23048
  method_id: pearson_corr
  metric_ids:
  • vc_grn
  • vc_hvg
  • vc
    metric_values:
  • '0.6184723973274231'
  • '0.6489825248718262'
  • '0.6337274610996246'
• dataset_id: op
  date_created: 17-12-2025
  file_size: 23048
  method_id: pearson_corr
  metric_ids:
  • rc_tf_act
    metric_values:
  • '0.35853068099355406'

In addition, we replicated the Pearson correlation–based evaluation. However, although TF binding precision, recall, and F1 score are intended to reflect transcription factor binding relevance, none of the reproduced values match the leaderboard result (reported as 0.09). Specifically, we obtained TF binding precision = 0.0478, recall = 0.0276, and F1 score = 0.0350.

[janursa]

your observation is correct that there are multiple sub metrics are outputted by metrics but only one or two are included in the leaderboard.
i will push a new update by tomorrow that lists which metrics are among the final ones.
i will also take a look at the issue with the TF binding scores.

[janursa]

@ZEXUANW i made an update to the documentation as well as the main repo. pls check and let me know if you still find something unclear

[ZEXUANW]

Hi @janursa ,

I have a couple of questions about how the leaderboard in the documentation is generated and how it corresponds to the outputs of the replication code:

https://genernib-documentation.readthedocs.io/en/latest/leaderboard.html

Regression (Pearson correlation, VirtualCell example)
On the leaderboard, the regression Pearson correlation for VirtualCell is reported as 0.69.
However, in my replication run I get three different entries:

vc_grn: 0.6184723973274231

vc_hvg: 0.6489825248718262

vc: 0.6337274610996246

None of these matches 0.69, and it’s not obvious which of them (or what combination of them) corresponds to the leaderboard value. Has the replication code or naming convention changed, or is the leaderboard using a different variant/aggregation?

SEM variants
Similarly, for SEM I see three variants in my outputs:

sem_grn: 0.0987688622048502

sem_hvg: 0.07334421631793887

sem: 0.08605653926139453

In the documentation, only “SEM” is reported with a single number. Which of these variants is used for the leaderboard, and how is the choice made?

Missing replica consistency and multi‑metric tasks

In my replication results I don’t see a separate “replica consistency” metric. In addition, some tasks produce multiple metrics, but only one value appears in the documentation. For example:

TF binding:

tfb_precision

tfb_recall

tfb_f1

rc_tf_act

Gene set recovery:

gs_precision: 0.70174168789988

gs_recall: 0.3279813455378404

gs_f1: 0.40692143797401853

gs_n_active: 348.6666666666667

On the leaderboard, only a single value is shown for each of these tasks. Could you clarify:

Which specific metric is used for each task on the leaderboard (e.g. F1 vs precision vs recall)?

Whether there is a mapping table or documentation that explains how the replication outputs (like vc_grn, sem_hvg, tfb_f1, gs_f1, etc.) are converted into the single numbers shown on the leaderboard?

Right now it’s a bit hard to match the replication outputs to the documentation, so any explanation of the naming/mapping or where to find that information would be very helpful.

Thanks a lot for clarifying!

[janursa]

@ZEXUANW metrics mapping is in surrogate_names (see https://genernib-documentation.readthedocs.io/en/latest/evaluation.html#).

pls pull the resources from aws. there might be a minor changes.

if you still dont reproduce the leaderboard, pls report so i take a look.
thanks