Evaluation Code for Models

[SdwHorizon]

Hello,

First of all, thank you for sharing this fantastic work! I was reviewing Table 4 in the paper, and I noticed the set of models highlighted in the green box (Qwen2.5-7B-Instruct, LLaMA-3.1-70B-Instruct, LLaMA-3.1-8B-Instruct, Mistral-7B-Instruct, Mixtral-8x22B-Instruct, Gemma-2-27B-Instruct).

I would like to clarify whether the evaluation code used for these models has been open-sourced. If it has not been released yet, do you have any plans to make it publicly available in the near future?

Having access to this code would be extremely helpful for understanding and reproducing the evaluation results for these models.

Thank you very much for your time and help!

Best regards,

[SdwHorizon]

[ViktorAxelsen]

Thank you for your interest in our work!

For fair comparison, all router baselines in this work generate the final answer using a shared base model (i.e., Qwen2.5-3B-Instruct or LLaMA-3.2-3B-Instruct). The models you highlighted in Table 4 are not evaluated as standalone generators. Instead, each individual model is first prompted to produce an intermediate response containing relevant information for answering the question. These intermediate outputs are then fed into the base model, which generates the final answer used for evaluation and assessment. This pipeline is consistent with all other router baselines to ensure fairness.

[jiayuww]

Hi, I’m a bit confused about how the cost in Table 4 is calculated. Is it the dollar cost computed as the number of completion tokens for the selected model times the API price per million tokens in route_service.py? And is this reported as the per-query cost or the total cost over all samples in the test set? I’m asking because based on the configuration, the completion token budget is 600. For example, if the test uses Llama-70B (cost = 0.88), then assuming 500 samples each using the full 600 tokens, the cost should be: 600 × 0.88 × 500 / 1,000,000 = $0.264, which is quite different in scale from the reported value of 153.3. Could you elaborate on how the cost is calculated in the paper? Thank you!

[ViktorAxelsen]

Hi, I’m a bit confused about how the cost in Table 4 is calculated. Is it the dollar cost computed as the number of completion tokens for the selected model times the API price per million tokens in route_service.py? And is this reported as the per-query cost or the total cost over all samples in the test set? I’m asking because based on the configuration, the completion token budget is 600. For example, if the test uses Llama-70B (cost = 0.88), then assuming 500 samples each using the full 600 tokens, the cost should be: 600 × 0.88 × 500 / 1,000,000 = $0.264, which is quite different in scale from the reported value of 153.3. Could you elaborate on how the cost is calculated in the paper? Thank you!

Thank you for the question!

We would like to clarify that the “raw cost reward” reported in Table 4 is not the actual dollar cost. Instead, it is computed as:

(API price per million tokens) × (number of generated completion tokens), averaged over all test samples

Importantly: The cost is averaged per query, not summed over the entire test set. We intentionally do not divide by 1,000,000, since doing so would make the values very small and inconvenient for comparison. The reported numbers are therefore scaled cost values, used only for relative comparison across baselines, not for billing or real monetary interpretation.

[jiayuww]

Thank you for your response! Could you please elaborate on how the baseline EM and cost are calculated?

Per your previous explanation, "Instead, each individual model is first prompted to produce an intermediate response containing relevant information for answering the question. These intermediate outputs are then fed into the base model, which generates the final answer used for evaluation and assessment", I would like to confirm whether my understanding is correct.

Specifically, for the KNN router, is the baseline implemented by first computing similarity between the test query and the training set, and then obtaining the corresponding full traces using the same model pool, router, and prompt (i.e., are these traces generated via the router’s own model selections, or are ground-truth models used)? These full (or partial?) traces (e.g., from the top-3 nearest neighbors) are then provided as context together with the new query, and the router model is asked to do the same search and produce the final answer. Or is a different procedure used?

Since this setup differs from the traditional evaluation setting, I would greatly appreciate a more detailed explanation of the exact process.

[ViktorAxelsen]

Thank you for your response! Could you please elaborate on how the baseline EM and cost are calculated?

Per your previous explanation, "Instead, each individual model is first prompted to produce an intermediate response containing relevant information for answering the question. These intermediate outputs are then fed into the base model, which generates the final answer used for evaluation and assessment", I would like to confirm whether my understanding is correct.

Specifically, for the KNN router, is the baseline implemented by first computing similarity between the test query and the training set, and then obtaining the corresponding full traces using the same model pool, router, and prompt (i.e., are these traces generated via the router’s own model selections, or are ground-truth models used)? These full (or partial?) traces (e.g., from the top-3 nearest neighbors) are then provided as context together with the new query, and the router model is asked to do the same search and produce the final answer. Or is a different procedure used?

Since this setup differs from the traditional evaluation setting, I would greatly appreciate a more detailed explanation of the exact process.

Thank you for the detailed question.

• General evaluation setup (shared by all router baselines, also described briefly in Sec 4.2 in the paper):

The goal of all router baselines in this work is the same: given a user query, select an appropriate LLM from the candidate pool to provide useful information for answering that query. For every test query, the evaluation proceeds as follows:

1. A (trained) router (e.g., KNN Router, RouterDC, etc.) selects one candidate LLM from the pool.

2. The selected LLM is prompted to produce intermediate information relevant to the query.

3. This intermediate output is then fed into the base model (e.g., Qwen or LLaMA), which generates the final answer.

4. The base model outputs the answer inside tags, which we parse and compare against the ground-truth answer to compute EM.

• KNN Router as an example

For the KNN Router specifically, the KNN model is first fit on the training set. At inference time, for each test query, the fitted KNN selects one candidate LLM from the pool. That selected LLM is then used only to generate intermediate information for the test query, following the same pipeline as other routers. Training examples or traces are NOT used as in-context examples during inference. Training data is used only to train the KNN router, not to provide retrieval-based context for the test query.

• Cost calculation

We compute cost only for the selected candidate LLM (the one producing intermediate output). The cost of the base model is ignored, since it is much smaller than the candidate models and negligible under our setup.

Across all router baselines, the only difference is which candidate LLM is selected for each test query. The downstream answer generation, EM evaluation, and cost calculation are otherwise identical.

I hope this clarifies the process.

[jiayuww]

Thank you for your response! Could you please elaborate on how the baseline EM and cost are calculated?
Per your previous explanation, "Instead, each individual model is first prompted to produce an intermediate response containing relevant information for answering the question. These intermediate outputs are then fed into the base model, which generates the final answer used for evaluation and assessment", I would like to confirm whether my understanding is correct.
Specifically, for the KNN router, is the baseline implemented by first computing similarity between the test query and the training set, and then obtaining the corresponding full traces using the same model pool, router, and prompt (i.e., are these traces generated via the router’s own model selections, or are ground-truth models used)? These full (or partial?) traces (e.g., from the top-3 nearest neighbors) are then provided as context together with the new query, and the router model is asked to do the same search and produce the final answer. Or is a different procedure used?
Since this setup differs from the traditional evaluation setting, I would greatly appreciate a more detailed explanation of the exact process.

Thank you for the detailed question.

• General evaluation setup (shared by all router baselines, also described briefly in Sec 4.2 in the paper):

The goal of all router baselines in this work is the same: given a user query, select an appropriate LLM from the candidate pool to provide useful information for answering that query. For every test query, the evaluation proceeds as follows:

1. A (trained) router (e.g., KNN Router, RouterDC, etc.) selects one candidate LLM from the pool.
2. The selected LLM is prompted to produce intermediate information relevant to the query.
3. This intermediate output is then fed into the base model (e.g., Qwen or LLaMA), which generates the final answer.
4. The base model outputs the answer inside tags, which we parse and compare against the ground-truth answer to compute EM.

• KNN Router as an example

For the KNN Router specifically, the KNN model is first fit on the training set. At inference time, for each test query, the fitted KNN selects one candidate LLM from the pool. That selected LLM is then used only to generate intermediate information for the test query, following the same pipeline as other routers. Training examples or traces are NOT used as in-context examples during inference. Training data is used only to train the KNN router, not to provide retrieval-based context for the test query.

• Cost calculation

We compute cost only for the selected candidate LLM (the one producing intermediate output). The cost of the base model is ignored, since it is much smaller than the candidate models and negligible under our setup.

Across all router baselines, the only difference is which candidate LLM is selected for each test query. The downstream answer generation, EM evaluation, and cost calculation are otherwise identical.

I hope this clarifies the process.

Thank you for your response! I was also wondering what query is used to obtain the intermediate information. Is it the same as the original question, or a different query that is not necessarily intended to produce the final answer? In other words, does the intermediate query need to align directly with the original question, or can it be any query used to gather related information?

[ViktorAxelsen]

Thank you for your response! Could you please elaborate on how the baseline EM and cost are calculated?
Per your previous explanation, "Instead, each individual model is first prompted to produce an intermediate response containing relevant information for answering the question. These intermediate outputs are then fed into the base model, which generates the final answer used for evaluation and assessment", I would like to confirm whether my understanding is correct.
Specifically, for the KNN router, is the baseline implemented by first computing similarity between the test query and the training set, and then obtaining the corresponding full traces using the same model pool, router, and prompt (i.e., are these traces generated via the router’s own model selections, or are ground-truth models used)? These full (or partial?) traces (e.g., from the top-3 nearest neighbors) are then provided as context together with the new query, and the router model is asked to do the same search and produce the final answer. Or is a different procedure used?
Since this setup differs from the traditional evaluation setting, I would greatly appreciate a more detailed explanation of the exact process.

Thank you for the detailed question.

• General evaluation setup (shared by all router baselines, also described briefly in Sec 4.2 in the paper):

The goal of all router baselines in this work is the same: given a user query, select an appropriate LLM from the candidate pool to provide useful information for answering that query. For every test query, the evaluation proceeds as follows:

1. A (trained) router (e.g., KNN Router, RouterDC, etc.) selects one candidate LLM from the pool.
2. The selected LLM is prompted to produce intermediate information relevant to the query.
3. This intermediate output is then fed into the base model (e.g., Qwen or LLaMA), which generates the final answer.
4. The base model outputs the answer inside tags, which we parse and compare against the ground-truth answer to compute EM.

• KNN Router as an example

For the KNN Router specifically, the KNN model is first fit on the training set. At inference time, for each test query, the fitted KNN selects one candidate LLM from the pool. That selected LLM is then used only to generate intermediate information for the test query, following the same pipeline as other routers. Training examples or traces are NOT used as in-context examples during inference. Training data is used only to train the KNN router, not to provide retrieval-based context for the test query.

• Cost calculation

We compute cost only for the selected candidate LLM (the one producing intermediate output). The cost of the base model is ignored, since it is much smaller than the candidate models and negligible under our setup.
Across all router baselines, the only difference is which candidate LLM is selected for each test query. The downstream answer generation, EM evaluation, and cost calculation are otherwise identical.
I hope this clarifies the process.

Thank you for your response! I was also wondering what query is used to obtain the intermediate information. Is it the same as the original question, or a different query that is not necessarily intended to produce the final answer? In other words, does the intermediate query need to align directly with the original question, or can it be any query used to gather related information?

For single-round routers, the query used to obtain intermediate information is the same as the original user question. The selected candidate LLM is directly prompted with the original query to gather relevant information.

For multi-round routers (e.g., KNN Router* and Prompt LLM*), the process is slightly different. In these cases, we first prompt the base LLM to decompose the original question into multiple sub-queries. Each sub-query is then assigned to a candidate LLM using the corresponding router (KNN Router or Prompt LLM). The intermediate outputs from these sub-queries are aggregated and fed back into the base model to produce the final answer.

[SdwHorizon]

Thank you for your interest in our work!

For fair comparison, all router baselines in this work generate the final answer using a shared base model (i.e., Qwen2.5-3B-Instruct or LLaMA-3.2-3B-Instruct). The models you highlighted in Table 4 are not evaluated as standalone generators. Instead, each individual model is first prompted to produce an intermediate response containing relevant information for answering the question. These intermediate outputs are then fed into the base model, which generates the final answer used for evaluation and assessment. This pipeline is consistent with all other router baselines to ensure fairness.

Thank you for your response!
Is this part of the evaluation code open source? If it is not possible to open-source, is it feasible to conduct a bare model evaluation of the model using the opencompass or lm_evl assessment tools?

[ViktorAxelsen]

Thanks for the question!

Since this work focuses on router-style training and evaluation, using a framework that is specifically designed for LLM routing will be more suitable than general-purpose evaluation tools.

Fortunately, we have recently open-sourced LLMRouter, which provides a unified framework for training and evaluating multiple classical router baselines under the same setup. It is fully aligned with the evaluation protocol used in this work and should be convenient for reproducing and extending the results.

Thanks again for your interest!