Open
Conversation
ElliottKasoar
added
the
new benchmark
joehart2001
reviewed
Mar 20, 2026
| Computational cost | ||
| ------------------ | ||
|
|
||
| Medium: benchmark involves only single-point calculations, but for 36,718 slab-bulk pairs. |
Collaborator
There was a problem hiding this comment.
it would be good to give a rough indication of time e.g. hours on gpu and minutes on gpu?
joehart2001
reviewed
Mar 22, 2026
| from pathlib import Path | ||
| from typing import Any | ||
|
|
||
| from ase.io import read |
Collaborator
There was a problem hiding this comment.
Suggested change
| from ase.io import read | |
| from ase.io import read, write | |
| from tqdm import tqdm |
joehart2001
reviewed
Mar 22, 2026
| / "cleavage_energy" | ||
| ) | ||
|
|
||
| results = {} |
Collaborator
There was a problem hiding this comment.
Suggested change
| results = {} | |
| write_dir = OUT_PATH / model_name | |
| write_dir.mkdir(parents=True, exist_ok=True) |
joehart2001
reviewed
Mar 22, 2026
|
|
||
| results = {} | ||
|
|
||
| for mpid_dir in sorted(d for d in data_dir.iterdir() if d.is_dir()): |
Collaborator
There was a problem hiding this comment.
Suggested change
| for mpid_dir in sorted(d for d in data_dir.iterdir() if d.is_dir()): | |
| idx = 0 | |
| for mpid_dir in tqdm(sorted(d for d in data_dir.iterdir() if d.is_dir())): |
joehart2001
reviewed
Mar 22, 2026
Comment on lines
+61
to
+76
| unique_id = slab.info["unique_id"] | ||
| results[unique_id] = { | ||
| "slab_energy": slab_energy, | ||
| "bulk_energy": bulk_energy, | ||
| "area_slab": float(slab.info["area_slab"]), | ||
| "thickness_ratio": float(slab.info["thickness_ratio"]), | ||
| "ref_cleavage_energy": float(slab.info["ref_cleavage_energy"]), | ||
| "mpid": slab.info["mpid"], | ||
| "miller": slab.info["miller"], | ||
| "term": int(slab.info["term"]), | ||
| } | ||
|
|
||
| OUT_PATH.mkdir(parents=True, exist_ok=True) | ||
| output_file = OUT_PATH / f"{model_name}.json" | ||
| with open(output_file, "w", encoding="utf-8") as f: | ||
| json.dump(results, f) |
Collaborator
There was a problem hiding this comment.
Suggested change
| unique_id = slab.info["unique_id"] | |
| results[unique_id] = { | |
| "slab_energy": slab_energy, | |
| "bulk_energy": bulk_energy, | |
| "area_slab": float(slab.info["area_slab"]), | |
| "thickness_ratio": float(slab.info["thickness_ratio"]), | |
| "ref_cleavage_energy": float(slab.info["ref_cleavage_energy"]), | |
| "mpid": slab.info["mpid"], | |
| "miller": slab.info["miller"], | |
| "term": int(slab.info["term"]), | |
| } | |
| OUT_PATH.mkdir(parents=True, exist_ok=True) | |
| output_file = OUT_PATH / f"{model_name}.json" | |
| with open(output_file, "w", encoding="utf-8") as f: | |
| json.dump(results, f) | |
| slab.info.update( | |
| { | |
| "slab_energy": slab_energy, | |
| "bulk_energy": bulk_energy, | |
| "area_slab": float(slab.info["area_slab"]), | |
| "thickness_ratio": float(slab.info["thickness_ratio"]), | |
| "ref_cleavage_energy": float(slab.info["ref_cleavage_energy"]), | |
| "mpid": slab.info["mpid"], | |
| "miller": slab.info["miller"], | |
| "term": int(slab.info["term"]), | |
| } | |
| ) | |
| write(write_dir / f"{idx}.xyz", slab, format="extxyz") | |
| idx += 1 |
Collaborator
|
Hey @vue1999, thanks for the PR and its looking super good. A few things:
Let me know if you've got any ideas or are unsure about any of my suggestions, thanks! |
joehart2001
reviewed
Mar 22, 2026
Comment on lines
+8
to
+13
| RMSE: | ||
| good: 0.0 | ||
| bad: 10.0 | ||
| unit: meV/A^2 | ||
| tooltip: Root Mean Squared Error of cleavage energies | ||
| level_of_theory: PBE |
Collaborator
There was a problem hiding this comment.
Suggested change
| RMSE: | |
| good: 0.0 | |
| bad: 10.0 | |
| unit: meV/A^2 | |
| tooltip: Root Mean Squared Error of cleavage energies | |
| level_of_theory: PBE |
joehart2001
reviewed
Mar 22, 2026
Comment on lines
+11
to
+12
| from ml_peg.analysis.utils.decorators import build_table, plot_parity | ||
| from ml_peg.analysis.utils.utils import load_metrics_config, mae |
Collaborator
There was a problem hiding this comment.
Suggested change
| from ml_peg.analysis.utils.decorators import build_table, plot_parity | |
| from ml_peg.analysis.utils.utils import load_metrics_config, mae | |
| from ml_peg.analysis.utils.decorators import build_table, plot_density_scatter | |
| from ml_peg.analysis.utils.utils import ( | |
| load_metrics_config, | |
| mae, | |
| write_density_trajectories, | |
| ) |
joehart2001
reviewed
Mar 22, 2026
Comment on lines
+29
to
+65
|
|
||
| def _load_model_results(model_name: str) -> dict | None: | ||
| """ | ||
| Load the JSON energy results for a model, or None if absent. | ||
|
|
||
| Parameters | ||
| ---------- | ||
| model_name | ||
| Name of the model whose results file to load. | ||
|
|
||
| Returns | ||
| ------- | ||
| dict | None | ||
| Parsed JSON results dictionary, or None if the file does not exist. | ||
| """ | ||
| result_file = CALC_PATH / f"{model_name}.json" | ||
| if not result_file.exists(): | ||
| return None | ||
| with open(result_file, encoding="utf-8") as f: | ||
| return json.load(f) | ||
|
|
||
|
|
||
| def system_names() -> list[str]: | ||
| """ | ||
| Get list of system identifiers from calc outputs. | ||
|
|
||
| Returns | ||
| ------- | ||
| list[str] | ||
| Sorted list of unique_id values from the first model with results. | ||
| """ | ||
| for model_name in MODELS: | ||
| data = _load_model_results(model_name) | ||
| if data is not None: | ||
| return sorted(data.keys()) | ||
| return [] | ||
|
|
Collaborator
There was a problem hiding this comment.
Suggested change
| def _load_model_results(model_name: str) -> dict | None: | |
| """ | |
| Load the JSON energy results for a model, or None if absent. | |
| Parameters | |
| ---------- | |
| model_name | |
| Name of the model whose results file to load. | |
| Returns | |
| ------- | |
| dict | None | |
| Parsed JSON results dictionary, or None if the file does not exist. | |
| """ | |
| result_file = CALC_PATH / f"{model_name}.json" | |
| if not result_file.exists(): | |
| return None | |
| with open(result_file, encoding="utf-8") as f: | |
| return json.load(f) | |
| def system_names() -> list[str]: | |
| """ | |
| Get list of system identifiers from calc outputs. | |
| Returns | |
| ------- | |
| list[str] | |
| Sorted list of unique_id values from the first model with results. | |
| """ | |
| for model_name in MODELS: | |
| data = _load_model_results(model_name) | |
| if data is not None: | |
| return sorted(data.keys()) | |
| return [] |
Comment on lines
+96
to
+105
| @plot_parity( | ||
| filename=OUT_PATH / "figure_cleavage_energies.json", | ||
| title="Cleavage Energies", | ||
| x_label="Predicted cleavage energy / meV/\u00c5\u00b2", | ||
| y_label="Reference cleavage energy / meV/\u00c5\u00b2", | ||
| hoverdata={ | ||
| "System": system_names(), | ||
| }, | ||
| ) | ||
| def cleavage_energies() -> dict[str, list]: |
Collaborator
There was a problem hiding this comment.
Suggested change
| @plot_parity( | |
| filename=OUT_PATH / "figure_cleavage_energies.json", | |
| title="Cleavage Energies", | |
| x_label="Predicted cleavage energy / meV/\u00c5\u00b2", | |
| y_label="Reference cleavage energy / meV/\u00c5\u00b2", | |
| hoverdata={ | |
| "System": system_names(), | |
| }, | |
| ) | |
| def cleavage_energies() -> dict[str, list]: | |
| def cleavage_energies() -> dict[str, dict[str, list]]: |
Comment on lines
+113
to
+114
| dict[str, list] | ||
| Dictionary of reference and predicted cleavage energies in meV/A^2. |
Collaborator
There was a problem hiding this comment.
Suggested change
| dict[str, list] | |
| Dictionary of reference and predicted cleavage energies in meV/A^2. | |
| dict[str, dict[str, list]] | |
| Dictionary of model names to ``{"ref": [...], "pred": [...]}`` in meV/A^2. |
Comment on lines
+116
to
+117
| results = {"ref": []} | {mlip: [] for mlip in MODELS} | ||
| canonical_ids = None |
Collaborator
There was a problem hiding this comment.
Suggested change
| results = {"ref": []} | {mlip: [] for mlip in MODELS} | |
| canonical_ids = None | |
| results = {mlip: {"ref": [], "pred": []} for mlip in MODELS} | |
| ref_stored = False | |
| stored_ref = [] |
Comment on lines
+120
to
+121
| data = _load_model_results(model_name) | ||
| if data is None: |
Collaborator
There was a problem hiding this comment.
Suggested change
| data = _load_model_results(model_name) | |
| if data is None: | |
| model_dir = CALC_PATH / model_name | |
| if not model_dir.exists(): |
|
|
||
| from ml_peg.app import APP_ROOT | ||
| from ml_peg.app.base_app import BaseApp | ||
| from ml_peg.app.utils.load import read_plot |
Collaborator
There was a problem hiding this comment.
Suggested change
| from ml_peg.app.utils.load import read_plot | |
| from ml_peg.app.utils.build_callbacks import plot_from_table_cell, struct_from_scatter | |
| from ml_peg.app.utils.load import collect_traj_assets, read_density_plot_for_model |
| """Cleavage energy benchmark app layout and callbacks.""" | ||
|
|
||
| def register_callbacks(self) -> None: | ||
| """No interactive callbacks needed.""" |
Collaborator
There was a problem hiding this comment.
Suggested change
| """No interactive callbacks needed.""" | |
| """Register callbacks to app.""" | |
| density_plots: dict[str, dict] = {} | |
| for model in MODELS: | |
| density_graph = read_density_plot_for_model( | |
| filename=DATA_PATH / "figure_cleavage_energies.json", | |
| model=model, | |
| id=f"{BENCHMARK_NAME}-{model}-density", | |
| ) | |
| if density_graph is not None: | |
| density_plots[model] = {"MAE": density_graph} | |
| plot_from_table_cell( | |
| table_id=self.table_id, | |
| plot_id=f"{BENCHMARK_NAME}-figure-placeholder", | |
| cell_to_plot=density_plots, | |
| ) | |
| struct_trajs = collect_traj_assets( | |
| data_path=DATA_PATH, | |
| assets_prefix="assets/surfaces/cleavage_energy", | |
| models=MODELS, | |
| traj_dirname="density_traj", | |
| suffix=".extxyz", | |
| ) | |
| for model in struct_trajs: | |
| struct_from_scatter( | |
| scatter_id=f"{BENCHMARK_NAME}-{model}-density", | |
| struct_id=f"{BENCHMARK_NAME}-struct-placeholder", | |
| structs=struct_trajs[model], | |
| mode="traj", | |
| ) |
Comment on lines
+38
to
+41
| scatter = read_plot( | ||
| DATA_PATH / "figure_cleavage_energies.json", | ||
| id=f"{BENCHMARK_NAME}-figure", | ||
| ) |
Collaborator
There was a problem hiding this comment.
Suggested change
| scatter = read_plot( | |
| DATA_PATH / "figure_cleavage_energies.json", | |
| id=f"{BENCHMARK_NAME}-figure", | |
| ) |
| docs_url=DOCS_URL, | ||
| table_path=DATA_PATH / "cleavage_energy_metrics_table.json", | ||
| extra_components=[ | ||
| Div(scatter, style={"marginTop": "20px"}), |
Collaborator
There was a problem hiding this comment.
Suggested change
| Div(scatter, style={"marginTop": "20px"}), | |
| Div(id=f"{BENCHMARK_NAME}-figure-placeholder"), | |
| Div(id=f"{BENCHMARK_NAME}-struct-placeholder"), |
| full_app = Dash( | ||
| __name__, | ||
| assets_folder=DATA_PATH.parent.parent, | ||
| suppress_callback_exceptions=True, |
Collaborator
There was a problem hiding this comment.
Suggested change
| suppress_callback_exceptions=True, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-review checklist for PR author
PR author must check the checkboxes below when creating the PR.
Summary
Adding a benchmark to evaluate the accuracy of predicting cleavage energies of crystalline surfaces.
Linked issue
Resolves #403
Progress
Testing
MACE omat, ORB v3
New decorators/callbacks
no