Language
Country/Area
No result found
The secrets of black box AI systems: How do they make decisions?
With the rapid development of artificial intelligence technology, more and more enterprises and government agencies rely on these systems to assist in making critical decisions. However, these AI systems are often called "black boxes" because even their designers cannot easily explain why they make certain decisions. This is expected to change with the rise of research into interpretive artificial intelligence (XAI), as researchers explore how to uncover the reasoning behind AI algorithms.
The purpose of interpretive AI is to increase users’ trust in AI systems so that they can use these technologies more effectively.
The basic concept of interpretive AI is to provide end users with the ability to understand how the AI makes decisions. The main focus in this area is to clarify the decision-making process of AI systems to enable users to assess their safety and trace the reasons behind automated decisions. In short, XAI hopes to enable people to understand the operating logic of AI systems by enhancing transparency and explainability.
The emergence of XAI is designed to respond to users’ doubts and concerns about the AI decision-making process. Especially in sensitive industries such as medical, finance, and law, it is crucial to understand the basis for AI decision-making.
Black box models are difficult to explain and even domain experts have trouble understanding their inner workings.
In artificial intelligence, machine learning algorithms are usually classified into white box models and black box models. The decision-making process of the white box model is transparent and easy for experts to understand; while the black box model is extremely difficult to explain, and even designers find it difficult to conduct reasonable analysis. The research goal of XAI is to improve the interpretability of these black box models to provide users with information to help them review and challenge AI decisions when necessary.
Interpretability, explainability and transparency
Interpretability can be understood through three principles: transparency, explainability, and interpretability. Transparency refers to the designer's ability to describe and explain the process of extracting model parameters from training data; Interpretability is the ability for users to understand the model output and provide a basis for decision-making; Interpretability covers the solutions used to formulate decisions characteristics.
If an algorithm can satisfy these principles, then it can provide reasonable explanations for decisions and be trusted by users.
Interpretability and explainability technology
When discussing the interpretability of an AI model, there are often specific techniques used to analyze the model's operation. For example, partial dependence plots can show the marginal impact of input features on prediction results; SHAP values help users visualize the contribution of each input feature to the model output.
There are a variety of other techniques for different types of models, such as Locally Interpretable Modeling (LIME), which provides an acceptable means of explanation by locally explaining the output of the current model with a simple model.
Using interpretive techniques can help ensure that AI models don’t make decisions based on irrelevant or unfair criteria.
The development history of XAI
Between 1970 and 1990, there were symbolic reasoning systems such as MYCIN and SOPHIE that were able to explain their reasoning processes. MYCIN is an early diagnostic system that interprets the hand-coded rules underlying its diagnosis, particularly for applications in the medical field.
XAI emerged in the 21st century as concerns about bias in AI and transparency in decision-making processes increased. Many academic institutions and businesses are beginning to focus on developing tools to detect potential bias and unfairness in their systems.
Regulation and Challenges
As regulators and general users increase their reliance on AI systems, the need for transparent, automated decision-making processes becomes increasingly urgent. For example, the European Union introduced a “right to interpretation” in its General Data Protection Regulation (GDPR), aiming to address potential issues arising from the growing influence of algorithms.
However, despite growing research on interpretive AI, these systems still face various challenges, such as malicious parties that may exploit weaknesses in interpretive technologies to achieve their own ends.
When we enjoy the convenience brought by AI, should we also be wary of the uncertainty of its internal operations?
