Explainable AI: Transparency and Accountability in ML, Including ChatGPT

The WHY?

Artificial intelligence (AI) is a powerful technology that can perform tasks that normally require human intelligence, such as understanding natural language, recognizing images, playing games, and generating content. However, many AI systems are based on complex algorithms that are difficult to understand and explain, especially for non-experts. These systems are often referred to as “black boxes”, because their inner workings are hidden from view.

This lack of transparency and interpretability can pose challenges for users, developers, regulators, and society at large. For example, how can we trust an AI system that makes decisions that affect our lives, such as diagnosing diseases, approving loans, or driving cars? How can we ensure that an AI system is fair, ethical, and accountable? How can we debug and improve an AI system that makes errors or behaves unexpectedly?

The WHAT?

These questions motivate the need for Explainable AI (XAI), a set of processes and methods that allows human users to comprehend and trust the results and output created by machine learning algorithms. Explainable AI is used to describe an AI model, its expected impact and potential biases. It helps characterize model accuracy, fairness, transparency and outcomes in AI-powered decision making.

Explainable AI is crucial for building trust and confidence in AI systems, especially in domains where human lives or values are at stake. It can also help developers improve and optimize their models, regulators ensure compliance with standards and regulations, and users understand and control their interactions with AI systems.

The WHEN?

The concept of XAI is not new. In fact, it has been around for decades. It was first introduced in 2004 by Van Lent et al.[1], in the context of the non-player characters [NPCs] in video games. The authors proposed a framework to extract key events and decision points from the playback log to allow the NPC AI controlled soldiers to explain their behavior in response to the questions selected from the XAI menu. Of course, this paper was not the first one to motivate this concept, but it was the first one to introduce the term “XAI” as per my knowledge.

The HOW?

Explainable AI can be achieved in different ways, depending on the type and complexity of the AI system, the level and purpose of the explanation, and the target audience. Some common approaches include:

Using interpretable models that are inherently transparent and easy to understand, such as decision trees, linear models, or rule-based systems. These are often called as Model-Specific approaches. These are tailored to a specific type of model and uses the characteristics of the model to provide explanations.
- Decision Trees: Decision trees are a type of model that can be easily visualized and interpreted. They provide explanations by showing the decision path from the root node to the leaf node.
- Rule-based Systems: Rule-based systems use a set of rules to make decisions. These rules can be easily interpreted and provide explanations by showing which rules were used to make a decision.
- Linear Models: Linear models are simple and interpretable, making them suitable for providing explanations. They provide explanations by showing the weights assigned to each feature in the input.
- Symbolic Models: Symbolic models use logical expressions to make decisions. They provide explanations by showing the logical rules used to make a decision.
Applying post-hoc techniques that analyze and explain the behavior of black-box models.These approaches provide explanations after the model has been trained and do not require any modification to the model architecture.
- Gradient-based methods: Gradient-based methods use the gradient of the model output with respect to the input to identify the important features. They provide explanations by highlighting the important features in the input.
- Perturbation-based methods: Perturbation-based methods involve modifying the input data and observing the changes in the output. They provide explanations by identifying which features in the input have the most impact on the output.
- Prototype-based methods: Prototype-based methods involve identifying the most representative instances in the data and using them to explain the model output. They provide explanations by showing which instances in the data are most similar to the input.
Providing interactive tools that allow users to explore and visualize the input-output relationships of AI systems. These are called Model-Agnostic Approaches since they do not require any specific knowledge about the underlying machine learning model and can be applied to any model. LIME and SHAP are examples of model-agnostic methods that provide local explanations of model predictions.
- LIME (Local Interpretable Model-Agnostic Explanations): LIME provides local explanations for individual predictions. It works by approximating the underlying model with a simpler, interpretable model in the local region around the prediction.
- SHAP (Shapley Additive exPlanations): SHAP provides global and local explanations for model predictions. It uses the concept of Shapley values from cooperative game theory to assign a contribution score to each feature in the input.
Incorporating human feedback and collaboration into the design and evaluation of AI systems, such as through user testing, co-creation, or explainable agents.
- User Testing: User testing involves asking users to perform tasks with an AI system and observing how they interact with it. This can be used to identify potential issues with the system and to improve the user experience.
- Co-creation: Co-creation involves involving users in the design and development of AI systems. This can be used to ensure that the system addresses the needs of the users and to improve the user experience.
- Explainable Agents: Explainable agents are AI systems that can explain their decisions to users. This can be used to improve the user experience and to build trust in the system.

The WHY NOW?

One domain where Explainable AI is particularly relevant is natural language processing (NLP), which is the branch of AI that deals with understanding and generating natural language. NLP has seen remarkable advances in recent years, thanks to the development of deep learning models such as ChatGPT.

ChatGPT is being used for countless applications, such as chatbots, text summarization, text generation, or text completion and so much more - ossinsight has a great list of ChatGPT applications and this is just the tip of the iceberg.

However, ChatGPT is also a black-box model that is hard to interpret and explain. It is not clear how ChatGPT generates its text outputs, what factors influence its choices, what biases or errors it might have, or how it can be improved or controlled. Moreover, ChatGPT can potentially generate harmful or misleading content that can affect users’ emotions, opinions, or actions.

In fact, these concerns are shared by leading tech luminaries such as Elon Musk, Steve Wozniak and others where they have signed an open letter title “Pause Giant AI Experiments: An Open Letter”. The letter calls for a moratorium on the development of AI systems that have “human-competitive intelligence” and “pose profound risks to society and humanity”. On a side note, this has been a very interesting topic of discussion and I recommend reading about it. Some of the good articles I found are:

In the light of all this, tools like Explainable AI can play an important role. By providing explanations for how ChatGPT works and why it produces certain outputs, Explainable AI can help users understand and trust ChatGPT’s capabilities and limitations. It can also help developers monitor and improve ChatGPT’s performance and quality. Furthermore, explainable AI can help regulators ensure that ChatGPT complies with ethical principles and social norms.

P.S. I am not an expert on XAI. This is just what I have been reading in the past few weeks. I am sure there are many more things to be said about XAI and its relevance in the modern AI research. I would love to hear your thoughts on this in the comments below.

References

[1] Van Lent, Michael, William Fisher, and Michael Mancuso. “An explainable artificial intelligence system for small-unit tactical behavior.” Proceedings of the national conference on artificial intelligence. Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999, 2004.

[2] Minh, Dang, et al. “Explainable artificial intelligence: a comprehensive review.” Artificial Intelligence Review (2022): 1-66.

[3] Schwalbe, Gesina, and Bettina Finzel. “A comprehensive taxonomy for explainable artificial intelligence: a systematic survey of surveys on methods and concepts.” Data Mining and Knowledge Discovery (2023): 1-59.

[4] Das, Arun, and Paul Rad. “Opportunities and challenges in explainable artificial intelligence (xai): A survey.” arXiv preprint arXiv:2006.11371 (2020).