Its obvious when a dog has been poorly trained. It doesnt respond properly to commands, pushes boundaries, and behaves unpredictably.The same is true with a poorly trained artificial intelligence (AI) model. Only with AI, its not always easy to identify what went wrong with the training.Research scientists globally are working with a variety of AI models trained on experimental and theoretical data. The goal is to predict a materials properties before creating and testing it. They are using AI to design better medicines and industrial chemicals in a fraction of the time it takes for experimental trial and error.But how can they trust the answers that AI models provide? Its not just an academic question. Millions of investment dollars can ride on whether AI model predictions are reliable.A research team from the Department of EnergysPacific Northwest National Laboratoryhas developed a method to determine how well a class of AI models called neural network potentials has been trained. Further, it can identify when a prediction is outside the boundaries of its training and where it needs more training to improvea process called active learning.The research team, led by PNNL data scientistsJenna BilbreyPope andSutanay Choudhury, describes how the new uncertainty quantification method worksin a research article published in NPJ Computational Materials.A dog that has been poorly trained is like an AI model that has been poorly trained. It doesnt know its boundaries. (Source: Jaromir Chalabala/Shutterstock) [Editors Note: But hes a good boy.]The research team, led by PNNL data scientistsJenna BilbreyPope andSutanay Choudhury, describes how the new uncertainty quantification method worksin a research article published in NPJ Computational Materials. The team is also makingthe method publicly available on GitHubas part of its larger repository, Scalable Neural Network Atomic Potentials (SNAP), to anyone who wants to apply it to their own work.We noticed that some uncertainty models tend to be overconfident, even when the actualerror in predictionis high, said Bilbrey Pope. This is common for most deep neural networks. However, a model trained with SNAP gives a metric that mitigates this overconfidence. Ideally, youd want to look atboth prediction uncertaintyand training data uncertainty to assess your overall model performance.Instilling trust in AI model training to speed discoveryResearch scientists want to take advantage of AIs speed of predictions, but right now, theres a tradeoff between speed and accuracy. An AI model can make predictions in seconds that might take a supercomputer 12 hours to compute using traditional computationally intensive methods. However, chemists and materials scientists still see AI as a black box.The PNNL data science teams uncertainty measurement provides a way to understand how much they should trust an AI prediction.AI should be able to accurately detect its knowledge boundaries, said Choudhury. We want our AI models to come with a confidence guarantee. We want to be able to make statements such as This prediction provides 85% confidence that catalyst A is better than catalyst B, based on your requirements.'In their published study, the researcherschose to benchmarktheir uncertainty method with one of the most advanced foundation models for atomistic materials chemistry, called MACE. The researchers calculated how well the model is trained to calculate the energy of specific families of materials. These calculations areimportantto understanding how well the AI model can approximate the more time- and energy-intensive methods that run on supercomputers. The results show what kinds of simulations can be calculated with confidence that the answers are accurate.This kind of trust and confidence in predictions is crucial to realizing the potential of incorporating AI workflows into everyday laboratory work and the creation of autonomous laboratories where AI becomes a trusted lab assistant, the researchers added.We have worked to make it possible to wrap any neural network potentials for chemistry into our framework, said Choudhury. Then in a SNAP, they suddenly have the power of being uncertainty aware.Now, if only puppies could be trained in a snap.In addition to Bilbrey and Choudhury, PNNL data scientistsJesun S. FirozandMal-Soon Leecontributed to the study. This work was supported by theTransferring exascale computational chemistry to cloud computing environment and emerging hardware technologies (TEC4) project, which is funded by the DOE Office of Science, Office of Basic Energy Sciences.About PNNLPacific Northwest National Laboratorydraws on its distinguishing strengths in chemistry, Earth sciences, biology and data science to advance scientific knowledge and address challenges inenergy resiliency and national security.Founded in 1965, PNNL is operated by Battelle and supported by the Office of Science of the U.S. Department of Energy. The Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of the most pressing challenges of our time. For more information, visit theDOE Office of Science website.For more information on PNNL, visitPNNLs News Center. Follow us onTwitter,Facebook,LinkedInandInstagram.Note: This article was initially posted on thePNNL News Siteand is reproduced here with permission.Karyn Hedeis aSenior Science Communicator and Media Relations Advisor at PNNL.