Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions docs/source/developer_guide/add_benchmarks.rst
Original file line number Diff line number Diff line change
Expand Up @@ -501,3 +501,44 @@ Building those components and their interactivity should become increasingly aut
but less standard plots/interactions will need setting up.

For now, please contact us to help with this process.

Framework credit tags
+++++++++++++++++++++

If a benchmark comes from an external benchmarking framework (for example,
MLIP Arena), add a framework credit tag as follows:

1. Add/update the framework entry in ``ml_peg/app/utils/frameworks.yml``.

.. code-block:: yaml

mlip_arena:
label: MLIP Arena
color: "#0f766e"
text_color: "#ecfeff"
url: "https://huggingface.co/spaces/atomind/mlip-arena"
logo: "https://huggingface.co/front/assets/huggingface_logo-noborder.svg"

2. Set ``framework_id`` in the benchmark app constructor.

.. code-block:: python3

return SomeBenchmarkApp(
name="SomeBenchmark",
...,
framework_id="mlip_arena",
)

That is all that is required. The benchmark header badge and the additional
framework pages for non-default frameworks are populated automatically from
this metadata.

Framework sections group matching benchmarks by category, omit the category
summary table, and reuse the same benchmark tables and controls. Updating
weights or thresholds there therefore updates the same benchmark views shown in
the category pages.

The ``logo`` field is optional. It can point to a remote image URL or a local
Comment thread
joehart2001 marked this conversation as resolved.
Dash asset path such as ``/assets/frameworks/my_framework_logo.png``. Use a
browser-supported image format such as ``.svg``, ``.png``, or ``.jpg``/``.jpeg``;
``.pdf`` is not supported. For best results, use a square logo image.
9 changes: 9 additions & 0 deletions ml_peg/app/base_app.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@

from ml_peg.app.utils.build_components import build_test_layout
from ml_peg.app.utils.load import rebuild_table
from ml_peg.app.utils.utils import normalize_framework_id


class BaseApp(ABC):
Expand All @@ -28,6 +29,9 @@ class BaseApp(ABC):
List of other Dash components to add to app.
docs_url
URL for online documentation. Default is None.
framework_id
Framework identifier used for benchmark attribution tags. Default is
``"ml_peg"``.
"""

def __init__(
Expand All @@ -37,6 +41,7 @@ def __init__(
table_path: Path,
extra_components: list[Component],
docs_url: str | None = None,
framework_id: str = "ml_peg",
):
"""
Initiaise class.
Expand All @@ -53,12 +58,15 @@ def __init__(
List of other Dash components to add to app.
docs_url
URL to online documentation. Default is None.
framework_id
Framework identifier used for benchmark attribution tags.
"""
self.name = name
self.description = description
self.table_path = table_path
self.extra_components = extra_components
self.docs_url = docs_url
self.framework_id = normalize_framework_id(framework_id)
self.table_id = f"{self.name}-table"
self.table = rebuild_table(
self.table_path, id=self.table_id, description=description
Expand All @@ -80,6 +88,7 @@ def build_layout(self) -> Div:
name=self.name,
description=self.description,
docs_url=self.docs_url,
framework_id=self.framework_id,
table=self.table,
thresholds=getattr(self.table, "thresholds", None),
extra_components=self.extra_components,
Expand Down
Loading
Loading