This excerpt is adapted from Trustworthy AI: A Business Guide for Navigating Trust and Ethics in AI by Beena Ammanath (Wiley, March 2022). Ammanath serves as executive director of the Global Deloitte AI Institute and heads Trustworthy & Ethical Technology at Deloitte. She has occupied leadership roles in artificial intelligence and data science at several companies, and she founded Humans For AI, an organization focused on expanding diversity in AI.
In AI model training, datasets stand in for the real world. Models are trained on one dataset and evaluated against another, and if the outcomes are alike, there is an assumption that the model’s functions can carry over to the operational setting. What succeeds in the lab ought to succeed consistently in the real world, but for how long? Ideal operating conditions are uncommon in AI, and real-world data is messy and intricate. This has given rise to what leading AI researcher Andrew Ng described as a “proof-of-concept-to-production gap,” where models train as intended but break down after deployment. It is in part an issue of robustness and reliability.
When outputs are not consistently accurate and worsen with time, uncertainty is the result. Data scientists are tasked with creating provably robust, consistently accurate AI models despite changing real-world data. In the flow of information, the algorithm can drift off course, with minor input changes producing major shifts in behavior.
To be sure, not every tool operates in settings exposed to dramatic change, and not every AI model carries the same degree of risk and consequence if it turns inaccurate or unreliable. As enterprises expand their AI presence, the challenge is to treat robustness and reliability as part of the AI strategy and to line up the processes, people, and technologies that can handle and correct errors in a shifting environment.
With that in mind, we begin with several of the core ideas in the field of robust and reliable AI.
Robust vs brittle AI
The International Organization for Standardization defines AI robustness as the “ability of an AI system to maintain its level of performance under any circumstances.” In a robust model, the training error rate, testing error rate, and operational error rate are all almost identical. And when unforeseen data appears in operation or when the model is running in less-than-ideal conditions, the robust AI tool keeps producing accurate outputs.
For instance, if a model can recognize every image of an airplane in a training dataset and has been shown to perform at a high level on testing data, then it ought to be able to identify airplane pictures in any dataset, even if it has not seen them before. But how does the airplane-recognition model do if a plane is pink, captured at dusk, missing a wing, or seen from an angle? Does its performance fall off, and if so, at what point is the model no longer workable?
When minor shifts in the environment produce major changes in functionality and accuracy, a model is described as inelastic or “brittle.” Brittleness is a familiar idea in software engineering, and it fits AI as well. In the end, all AI models are brittle to some extent. The different kinds of AI tools we use are tied to their function and their application. AI does only what we train it to do.
There is another element here. Those who deploy and manage AI have to consider how changing real-world data causes model accuracy to erode over time. In the phenomenon of “model drift,” the predictive accuracy of an AI tool declines as the underlying variables that inform the model shift. Signals and data sources that were once dependable can turn unreliable. Unexpected network failures can alter data flows.
An AI that plays chess is likely to stay robust over time, because the rules of chess and the moves the AI will face are predictable and fixed. By contrast, a natural language processing (NLP) chatbot works within the fluid setting of speech patterns, colloquial language, incorrect grammar and syntax, and a range of changing factors. With machine learning, unexpected data or incorrect computations can send a model off course, and what starts as a robust tool weakens into brittleness unless corrective tactics are used.
Developing reliable AI
The European Commission’s Joint Research Centre notes that assessing reliability calls for attention to performance and vulnerability. Reliable AI behaves as expected even with inputs that were not part of the training data, known as out-of-distribution (OOD) inputs. These are data points that differ from the training set, and reliable AI has to be able to determine whether data is OOD. One difficulty is that for some models, OOD inputs can still be labeled with high confidence, meaning the AI tool appears reliable when in fact it is not.
Consider an autonomous delivery robot. Its navigation AI is tuned to locate the shortest route to its destination. The training dataset includes all the sample data the AI needs to identify sidewalks, roads, crosswalks, curbs, pedestrians, and every other variable—except railroad tracks cutting across a path. In operation, the robot detects rail tracks in its way, and although they are OOD, the AI assigns high confidence to the idea that the tracks are merely a new kind of footpath, which it follows to speed up its delivery. Clearly, the AI has gone off course because of an OOD input. If it is not struck by a train, it confirms for the delivery robot, “this is a viable path” and may search for other rail tracks to use. And the operators may remain unaware – until a train comes along.
Reliable AI is accurate when faced with any novel input. This is not the same as average performance. A model that has good average performance may still produce occasional outputs with major consequences, undermining reliability. If an AI tool is accurate 80% of the time, is it a trustworthy model? A related issue is resilience to vulnerabilities, whether they arise naturally through operation or come from adversarial exploits.
Lessons in data reliability
A model is only as good as the training and testing data used to build it. Without confidence in the data quality vis-à-vis its representation of the real world, the model’s outputs may not reliably produce accurate outputs in the operational environment. For the U.S. Government Accountability Office, data reliability depends on:
- Applicability – Does the data provide valid measures of relevant qualities? Completeness – To what degree is the dataset populated across all attributes? Accuracy – Does the data reflect the real world from which the dataset was gathered?
These are cross-cutting elements of trustworthy data, as well as AI. Datasets need to be adequately curated and in some cases labeled or even supplemented with synthetic data, which can make up for missing data points or fill in for protected information that cannot (or should not) be used in training. Data also has to be scrubbed for latent bias, which skews model training and results in undesirable outputs or predictions.
As with the AI tool itself, real-world operational data needs to be watched for changing trends and emerging data science needs. For example, a model carrying out sentiment analysis may be trained to score sentiment across a dozen variables, but after deployment, the AI team identifies other variables that need to be considered in model drift and retraining.
Like reliability, data applicability is not fixed. Likewise, data accuracy may rise or fall depending on how well sensors perform, whether there are latency or availability problems, or any of the known factors that can impair data reliability.
Leading practices in building robust and reliable AI
Whether a model is hindered by unfamiliar data, disturbed by a malicious actor, or drifting away from accuracy, organizations should build into their AI initiatives the ability to assess deployment risk, monitor performance against intended specifications, judge (if not quantify) robustness, and have the processes in place to repair failing or drifting models as their reliability declines. Because reliability comes from robustness, some of the activities that can support AI reliability include:
Benchmarks for reliability
Even as model training continues, identify and define the benchmarks most useful for tracking and measuring reliability. These benchmarks may include how the AI system performs compared with human performance, which is especially apt because deep learning models seek to imitate human cognition overall in practice.
Perform data audits
As part of testing, review data reliability assessments, corrective actions, and training data samples. Bring in data stakeholders (e.g., IT leaders, legal experts, ethicists) to examine data quality and reliability. AI models need datasets that mirror the real world, so as part of data audits, assess the extent to which datasets are balanced, unbiased, applicable, and complete in practical terms here.
Monitor reliability over time
Reliability changes across the AI lifecycle. When model output or prediction departs from expectations, catalog the data for analysis and investigation. The data often used for this analysis includes time-to-event data (how long until the model diverged), degradation data (information surrounding how the model degrades), and recurrent events data (errors that occur more than once) during routine monitoring and operator review.
Uncertainty estimates
Insight builds confidence. To provide deeper visibility into how AI functions, emerging tools allow the model to report its degree of uncertainty alongside a prediction or output. This moves toward trust in robust systems. If a model reports high uncertainty, that is valuable insight for the human operator or another networked AI. Uncertainty estimates can flag a drifting model, highlight changes in data, or provide awareness that an adversarial example entered the data stream during live system operation.
Managing drift
Operators can assess drift by comparing the model’s inputs and outputs during live deployment with inputs and outputs in a reference set. Similarity is measured pairwise between test and training data inputs, with segmentation carried out on the outputs. By keeping a close understanding of how inputs and outputs change relative to the reference set, human operators are positioned to take corrective steps (e.g., retrain the model) as conditions and data distributions shift.
Continuous learning
Establish continuous learning workflows to monitor model performance against predefined acceptable thresholds. These thresholds might include measures of how resilient the system accuracy remains over time in the face of small perturbations, as well as safety constraints for the system and the environment in which it operates. As part of this, maintain a data version control framework to enable auditability, transparency, and reproducibility of the AI model.
Ongoing testing
Develop a testing regime that includes variability (e.g., changes in the system or training data) to evaluate whether AI is robust enough to function as intended. The frequency with which models are checked for robustness and accuracy should depend on the priority of the model and how often it is updated. High-risk, regularly updated models might best be checked daily (with a human verifying outputs). Slower changing, low-priority models could be checked on a longer timeline, in some cases using an API for automatic assessments of functionality. The results of these checks should prompt investigation and resolution of any exceptions, discrepancies, and unintended outcomes by responsible human reviewers.
Explore alternative approaches
Given that robustness and generalizability remain areas of active research, new tools, designs, and tactics will continue to appear and move the field forward. These are likely to be technical approaches, and the organization’s data science professionals are well placed to explore how new ideas can support deployed AI, as well as model development. For example, “Lipschitz constrained models” have bounded derivatives that can help neural networks become more robust against adversarial examples. Put simply, they encourage and can certify that small perturbations in input produce small changes in output.