One cornerstone of making AI work is machine learning – the ability for machines to learn from experience and data, and improve over time as they learn. In fact, it’s been the explosion in research and application of machine learning that’s made AI the hot bed of interest, investment, and application that it is today. Fundamentally, machine learning is all about giving machines lots of data to learn from, and using sophisticated algorithms that can generalize from that learning to data that the machine has never seen before. In this manner, the machine learning algorithm is the recipe that teaches the machine how to learn, and the machine learning model is the output of that learning that can then generalize to new data.
Regardless of the algorithm used to create the machine learning model, there is one fundamental truth: the machine learning model is only as good as its data. Bad data results in bad models. In many cases, these bad models are easy to spot since they perform poorly. For example, if you built a machine learning model to identify cats in images, if the model identifies butterflies as cats or fails to spot obvious cats in images, we know that there’s something wrong with the model.
There’s many reasons why a model could perform poorly. The input data could be riddled with errors or poorly cleansed. The various settings and configurations for the model (“hyperparameters”) could be set improperly yielding substandard results. Or perhaps the data scientists and ML engineers that trained the model selected a subset of available data that had some sort of inherent bias in it, resulting in skewed model results. Maybe the model just wasn’t trained enough, or had issues of overfitting or underfitting resulting in poor results. Indeed, there are actually many ways that a resulting model could be substandard.
What if instead of a cat classification model, we’ve built a facial recognition model, and we were using this model for security purposes. If the model misidentifies individuals, is the fault with improper configuration of the model, a poorly trained model, bad input data, or perhaps just we selected a biased set to train the model in the first place? If we are supposed to depend on this model, how can we trust that model knowing that there are so many ways for the model to fail?
The Problem of Transparency
In an typical application development project, we have quality assurance (QA) and testing processes, tools, and technologies that can quickly spot any bugs or deviations from established programming norms. We can run our applications through regression tests to make sure that new patches and fixes don’t cause more problems and we have ways to continuously test our capabilities as we continuously integrate them with increasingly more complex combinations of systems and application functionality.
But here is where we run into some difficulties with machine learning models. They’re not code per se in that we can’t just examine them to see where the bugs are. If we knew how the learning was supposed to work in the first place, well, then we wouldn’t need to train them with data would we? We’d just code the model from scratch and be done with it. But that’s not how machine learning models work. We derive the functionality of the model from the data and through use of algorithms that attempt to build the most accurate model we can from the data we have to generalize to data that the system has never seen before. We are approximating, and when we approximate we can never be exact. So, we can’t just bug fix our way to the right model. We can only iterate. Iterate with better data. Better configuration hyperparameters. Better algorithms. More horsepower. Those are the tools we have.
If you’re the builder of the model, then you have those tools at your disposal to make your models better. But what if you’re the user or consumer of that model? What if that model that you are using is performing poorly? You don’t have as much control over rebuilding that model to begin with. However, even more significantly, you don’t know why that model is performing poorly. Was it trained with bad data? Did the data scientists pick a selective or biased data set that doesn’t match your reality? Were the wrong hyperparameter settings chosen that might work well for the developers but not for you?
The problem is that you don’t know the answers to any of those questions. There is no transparency. As a model consumer, you just have the model. Use or it lose it. Your choice is to accept the model or go ahead and build your own. As the market shifts from model builders to model consumers, this is increasingly an unacceptable answer. The market needs more visibility and more transparency to be able to trust models that others are building. Should you trust that model that the cloud provider is providing? What about the model embedded in the tool you depend on? What visibility do you have to how the model was put together and how it will be iterated? The answer right now: little to none.
Problem #1: Unexplainable Algorithms
One issue that has been well acknowledged is that some machine learning algorithms are unexplainable. That is to say, when a decision has been made or the model has come to some conclusion such as a classification or a regression, there is little visibility into the understanding of how the model came to that conclusion. Deep learning neural networks, the current celebutante of machine learning these days particularly suffers from this problem. For example, when an image recognition model recognizes a turtle as a rifle, what is the reason why the model does so? We don’t actually know why, and therefore those looking to take advantage of model weaknesses can exploit such vulnerabilities by throwing these deep learning nets for a loop. While there are numerous efforts underway to add elements of explainability to deep learning algorithms, we are not quite yet at the point where such approaches are seeing widespread adoption.
Not all machine learning algorithms suffer from the same explainability issues. Decision trees by their nature are explainable, although when used in ensemble methods such as Random Forests, we lose elements of that explainability. So the first question any model consumer should ask is how explainable is the algorithm that was used to build the model? If we know what extent a model user can gain explainability for a given model decision, we can have some aspect of visibility into model performance.
Problem #2: Lack of visibility into training data sets
However, simply knowing if a given model uses an explainable algorithm is not enough to understand a model’s performance. As mentioned above, the strength of a model depends significantly on its training data. Good, clean, well-labeled (in the case of supervised learning approaches) data will result in good, well-performing models. Well… not always. If the training data doesn’t represent the data that you’re using in the real world, then even these highly-trained, cleaned, well-labeled models will perform poorly.
So, the next question that a model user will need to ask is about the training data. Where is that data from? How was it cleaned? What are the features or dimensions upon which the model was trained? Can I see the training data? Can I get visibility into how it was cleansed and what features were used? If the answer is no to these questions, then you have very limited visibility and you’re trusting that the model has your best intentions in mind.
Problem #3: Lack of visibility into methods of data selection
Let’s say you have access to the full set of training data used. This would represent the ultimate in transparency for the model right? Not quite. Just because you have access to the gigabytes or petabytes of data used to train the model doesn’t mean you know what aspects of that data was actually used to train the model. What if the ML engineers only chose to use a subset of that data, or specific dimensions, columns, or features of that data set? What if the data scientists used data augmentation approaches to enhance training data with additional data not in the training data set? Simply having access to the training data does not answer all questions of transparency.
Full transparency also means knowing how data was selected from the available training data, and ideally even being able to use the same selection methods on the training data yourself as a model consumer to see what data was included and what was excluded. You might realize that your use of the model is failing because the data you are using in real life just happens to be the data that was excluded from the data used for training.
Problem #4: Limited understanding of the bias in training data sets
Many times, models run into problems not because of bad data or even poorly selected data, but because there’s some inherent bias in the training data. The word “bias” is somewhat overloaded int he machine learning sense since we use it in three different ways: in the sense of the weights and “biases” set in a neural network, in the separate sense of the “bias”-variance tradeoff that balance overfit or underfit, and in the more widely understood sense of informational “bias” that is imposed by humans making decisions based on their own preconceived notions. It’s the last sense that we are most concerned.
There have been many issues with regards to facial recognition models trained on limited data sets due to the bias of model developers, and issues of loan decision models that have used historically biased data sets to determine availability of credit. Societal bias in data sets are significant and rife, and organizations need to find ways to eliminate such bias if they result in models that perpetuate biases in unwanted ways. This is a concept popularized by Cathy O’Neill in the book Weapons of Math Destruction.
However, this is only the most visible of biases. There are other biases that could just as much taint the usefulness of a given model. Perhaps an online ecommerce shop is using a model to automatically categorize and tag dresses for an online fashion store. If the model was trained using Western data, then wedding dresses would be categorized primarily by identifying white and similar shades as wedding dresses. But this model would fail in non-Western countries where colorful dresses, sarees, kimonos, and other forms of wedding dress are the accepted norms.
In other cases, insurance industries might be using models to identify and classify vehicles. Yet if these models were trained on well-lit, daytime photos of new vehicles fresh from the factory, the would fail terribly on photos of older, more beat-up versions of the same vehicles taken in the night time or in poor lighting conditions. The reality is that the model user doesn’t know the existing bias of the data set. It would be helpful to know that the inherent biases of the model might be. Perhaps the model developer doesn’t even know the biases, but the model users sure will find them out.
Problem #5: Limited visibility into model versioning
Another problem with model transparency is that models might be continuously versioned. Indeed, good models should be iterated on so that they can be even better. This is part of well-established methodologies. However, if you’re a model user and the model that you’re using starts to perform poorly when it was performing just fine before, you might be wondering why that is the case. The first instinct for model users that see sudden model performance issues is to look at the data that’s being fed into the model to see if anything has changed. But what would you think if the data is the same but the model performs poorly? Clearly something has changed in the model. This is the challenge of not having visibility into model versioning.
If you’re using a cloud-based model, seemingly random and spontaneous model versioning could cause problems. For full transparency, those producing models should also make it clear and consistent how models will be versioned, model versioning frequency, and the ability to use older model versions if new models start to perform poorly. While good model providers will provide full visibility into model versioning, this is not done in a consistent or guaranteed manner, especially when comparing between different model providers.
Other issues related to model trust and transparency
There are many other issues to consider with regards to model transparency that can cause model users to think twice before depending on a model for highly critical applications. Do you know why the model was built and what the model developers had in mind for its use case? Are you using the model the way the model builders intended? Was there any analysis done on the potential impact that the model might have for different users? What is the origin of the training data? What are the various performance metrics for different types of input data? How does this model perform in production on various measures? Can you apply transfer learning to the model, and if so how? How have model performance metrics changed over time? Who else is using the model? These are all valid and important questions to answer, with the more questions answered the greater the transparency for that give model.
A Proposed Transparency Assessment Method
Attempts at model transparency are not new. Google recently announced their Google Cloud Model Cards approach which addresses some of the considerations for model transparency, especially around performance and applicability to specific data sets. However, it does not address many of the aspects listed above, and for good reason. Their approach is primarily focused on helping the model builder build better models, but not necessarily focused on third party model consumers. While the Model Cards do have one aspects of “fairness” addressed in the approach, many of the measures above have not yet made it into the Model Cards specification.
In May 2019, the National Institute of Standards and Technology (NIST), a US Federal government agency, convened a meeting for the advancement of AI standards as per the White House’s AI strategy plans. At that meeting, the idea was brought up of a method by which models can be assessed for transparency across multiple measures. However, there was no action taken by NIST consequent to that. As a result, analysts from Cognilytica developed a multi-factor transparency assessment that was contributed to the non-profit organization ATARC, which subsequently picked it up for action in one of the Artificial Intelligence working groups. (Disclosure: I’m a principal analyst with Cognilytica and a working group chair at ATARC).
The ATARC AI Ethics and Responsible AI working group iterated on the proposal and produced a document that aims for model developers to assess their models on five factors for transparency. Each of the factors maps to one of the problem areas mentioned above, and assigns a point value for transparency along that measure. For example, a model could score a “1” for algorithmic explainability (meaning a total black box) while scoring a “5” for training data transparency (full access to the training data set is provided). The final result is a “radar chart” that shows the transparency for a given model as illustrated below:
Currently ATARC is looking to contribute the model as a standard to the International Organization for Standardization (ISO), and is also encouraging others to contribute and provide feedback. Regardless of whether this particular approach to transparency is broadly adopted, it’s hoped that large and small model builders alike are encouraged to include a model transparency assessment as part of their model release.
It’s important to note that model assessments are self-assessed, meaning that those who build models will produce the assessment for the models they build. While it is possible for model developers to inaccurately state their levels of transparency, it’s very easy for model users to verify and validate their self-assessments. After all, you can’t declare that model training data is available for anyone to access if it is not in fact the case.
A transparency assessment is sorely needed by the industry. In order to trust AI, organizations need models they can trust. It’s very hard to trust any model or any third party source without having transparency into how those models operate. While there are no doubt many organizations and standards bodies working on similar efforts to improve model transparency, there clearly needs to be some consensus in the industry on methods for transparency for the industry to see the long-term gains it hopes to achieve with AI.
Credit: Google News