Neural force fields: the field still has no single, agreed metric for whether they got it right. A short map of the mess — and where the real problems hide.
Structure relaxation — nudging a guessed arrangement of atoms downhill until the forces vanish — is the core inner loop of computational materials science. A machine-learned interatomic potential (MLIP) makes each step cheap, and modern potentials run this loop across enormous candidate libraries. Which raises a deceptively simple question: when the model hands back a “relaxed” structure, how do you score whether it succeeded?
Ask three groups and you may get three answers. One reports energy error against a reference calculation; one reports the mean distance from the true relaxed geometry; one runs a single expensive quantum calculation on the output and checks whether the forces really vanished. They are not measuring the same thing, and — as we will see — they do not always agree on who won.
The metrics in circulation sort cleanly into three families. Each answers a real question; each has a characteristic way of lying.
| Family | Typical metrics | What it measures — and its blind spot |
|---|---|---|
| Geometry | coordinate RMSD / MAE, max per-atom displacement, fraction of structures within a distance threshold (the OC20 IS2RS style) | How close the shape is to the reference. Blind spot: a small coordinate error need not sit at a true minimum, and a global mean is dominated by a handful of hard structures. |
| Relaxedness | residual force magnitude, max/mean force norm, fraction of structures with forces below a threshold (AFbT) | Whether the output is physically at equilibrium. Blind spot: forces can be small at the wrong minimum; and force error is often uncorrelated with the thing you care about. |
| Cost | DFT/optimizer steps saved, fraction reaching a convergence criterion | Whether the prediction actually cheapens the downstream calculation — the honest goal. Blind spot: expensive to measure, so it is reported least often, on the smallest samples. |
Two facts follow immediately. First, the metric you emphasise encodes what you think the task is — a shape, a stationary point, or a saved computation. Second, the cheapest metrics to compute (geometry, force error) are the furthest from the goal that actually matters (cost saved).
There is also a practical asymmetry the last two families hide. To score residual force or steps-saved honestly you need a fresh DFT calculation on the predicted structure — the model's own forces only certify that it reached its own fixed point, not a true minimum. And DFT is both non-differentiable (so it can't be the training loss) and slow, minutes per structure on several GPUs. So in day-to-day work the geometry family — comparing against the DFT-relaxed target the dataset already provides — is the default, with force and cost metrics reserved for a small validation subset. That is a hard constraint, not a stylistic choice.
The most common reporting choice — a single dataset-averaged error — is also the most treacherous, because relaxation datasets are heterogeneous. A few structures start far from equilibrium, or have many atoms, or are simply hard; they dominate the average and drown out the typical case.
We can put a number on it from our own work. On an oxide-surface relaxation set of 150 structures, the base potential's average structural error (a mass-weighted RMSD) has a mean-to-median ratio of about 1.3 — the mean sits a third above the typical structure, pulled up by the tail. Eighty-two percent of structures improve over doing nothing, but the median structure closes only ~9% of its distance to the target. Report only the mean and you would misread both the typical quality and the spread.
The fixes are known and cheap: report the median and the distribution, not just the mean; report the fraction of structures within a threshold (a first-class metric, not an afterthought); and, best of all, report per-structure relative improvement — how much of each structure's initial error the method removed — which normalises away the “a few hard structures” effect and answers the question collaborators actually ask: are we consistently less wrong after the model than before?
None of this is a scoop, and it would be dishonest to pretend otherwise. “Aggregate error metrics can mislead; use physics-, distribution- and task-aware evaluation instead” is by now a mature, actively-curated conversation. The honest thing is to name the people who built it.
Read together, these say the same thing from six directions: a scalar averaged error is a starting point, not a verdict.
So is there anything left to do, or is the topic closed? Being strict about it: most of the obvious moves are taken. A taxonomy of metrics is a blog post (you are reading one). A generic new scalar is unlikely to survive review next to the work above. The genuinely thin spots are narrow and specific:
— A metric validated against relaxation cost — the number of DFT or optimiser steps a prediction actually saves — rather than against molecular-dynamics observables, which is where the validated metrics currently live. And a demonstration that the metric choice reverses the ranking of named, state-of-the-art relaxation models on a real relaxation task, rather than on illustrative placeholders.
A word of caution against overselling, since it is tempting: the “mean hides the tail” effect is dilution, not a sign-reversing paradox; and “relaxation metrics are an unstudied gap” is false. The interesting residue is small, and it is shrinking as the benchmarks multiply.
The absence of a single yardstick is not a scandal to be fixed with one more scalar. It is the ordinary condition of a young field sitting on the seam between two older ones — and the durable contributions have been standardised benchmarks and validated metrics, not clever averages.
For those of us arriving from computer science, that is oddly good news. The “mess” — different data, different tasks, different notions of success — is exactly the surface where the interesting, well-posed problems live: what should we measure, how do we validate that a metric predicts the goal, and how do we compare methods without fooling ourselves. Materials science supplies the gold, which is the data. The measurement question is one place the two fields genuinely need each other.
The modest, honest version of all this, and the one we are adopting in our own relaxation work: stop reporting a lone mean. Show the median, the distribution, the fraction within threshold, and the per-structure improvement over the baseline — and cite the people who told us to.
Provenance. The mean-to-median figure is measured on a 150-structure oxide-surface relaxation set from our own experiments; the metric-family map and the open-problem assessment are synthesised from the six sources above, all linked inline. This is a survey and a position, not a new result. Companion: our G-DEQ work on relaxation training, whose evaluation motivated the question.
← All AI-OWLS posts