A NIST paper attempts to answer this question:
Briefly, our four principles of explainable AI are:
Explanation: Systems deliver accompanying evidence or reason(s) for all outputs.
Meaningful: Systems provide explanations that are understandable to individual users.
Explanation Accuracy: The explanation correctly reflects the system’s process for generating the output.
Knowledge Limits: The system only operates under conditions for which it was designed or when the system reaches a sufficient confidence in its output.
Four Principles of Explainable Artificial Intelligence
Stating this differently: there should be an explanation, it should be understandable and accurate, and the system should stop when it’s generating nonsense.
These are very reasonable principles, but likely tough to deliver with current technology.
Indeed, the paper discusses that humans are often unable to explain why they have taken a certain action:
People fabricate reasons for their decisions, even those thought to be immutable, such as personally held opinions [24, 34, 99]. In fact, people’s conscious reasoning that is able to be verbalized does not seem to always occur before the expressed decision. Instead, evidence suggests that people make their decision and then apply reasons for those decisions after the fact [95]. From a neuroscience perspective, neural markers of a decision can occur up to 10 seconds before a person’s conscious awareness [85]. This finding suggests that decision making processes begin long before our conscious awareness.
Id. at 14.
And it is well documented that even experts generally cannot predict their own accuracy.
What hope do the AI’s have?