In a chemical design review, “the model said so” is not a justification. Someone will ask for the mechanism, the domain of validity, and what happens outside the training set.
Where black boxes break
- Extrapolation — sparse corners of composition or temperature invent non-physical behaviour.
- Silent failure — no confidence band, no out-of-domain flag.
- No lever for chemists — you cannot challenge a weight matrix the way you challenge a mixing rule.
That is why physics-informed hybrids exist: keep a science baseline, learn residuals, show uncertainty.
What survives the room
Predictions that separate baseline, correction, and confidence give reviewers a place to push. If the residual dominates in a region with no data, the honest answer is “we do not know yet” — which is better than a precise wrong number.
AI that cannot be challenged will not be allowed next to capital, regulation, or plant safety cases. Transparency is not branding; it is the adoption requirement.
