Among Palo Alto’s machine-learning engineers, there is a somewhat awkward joke that goes around: the more impressive the model, the less anyone can explain how it truly functions. People chuckle before switching topics. It’s difficult to ignore how frequently that joke comes up in serious discussions, even when regulators are present, when observing the industry over the past few years.
It was promised that eventually these systems would open up and the gears would start to turn. Rather, the gears multiplied. Hundreds of billions of parameters can be present in a modern model, layered in a way that makes it difficult for human reasoning to understand. According to engineers, looking at activation maps is similar to looking at a far-off nebula; while patterns are obvious, meaning is more difficult.
| Field | Detail |
|---|---|
| Topic | Explainable AI (XAI) and the limits of interpretability |
| Featured Expert | Nigam Shah, Professor of Medicine and Biomedical Data Science |
| Affiliated Institution | Stanford University, Stanford Institute for Human-Centered AI |
| Core Problem | Deep neural networks behave as “black boxes,” with millions of parameters resisting human reading |
| Common Techniques | SHAP, LIME, saliency maps, surrogate models, concept bottleneck models |
| Industries Most Affected | Healthcare, finance, criminal justice, hiring, housing |
| Notable Case Study | A hospital mortality model flagged priest visits as a predictive factor |
| Regulatory Pressure | Rising in EU and US, especially for automated decisions on bail, loans, jobs |
| Underlying Tension | Predictive accuracy versus human-readable reasoning |
| Open Question | Whether full interpretability is even mathematically possible for the largest models |
Stanford professor Nigam Shah, who has been involved in this debate for years, makes a startling point. He points out that doctors don’t fully comprehend the mechanism underlying the majority of the approximately 4,900 medications that are regularly prescribed. Despite this, they continue to prescribe because research indicates that the medications are beneficial. He contends that AI in medicine could gain credibility in a similar manner through careful testing rather than philosophical openness. It’s a reasonable stance, but there’s something unsettling about it.
Because medicine is a different matter entirely. Job screenings, mortgage approvals, and bail hearings are all different. The lack of an explanation when a model silently refuses to give someone a loan is not a question. The issue is one of civil rights. Shah himself makes this distinction very evident: causal explanations are required in automated high-stakes decisions. Though slow is doing a lot of work in that sentence, regulators in Washington and Brussels appear to be gradually coming to the same conclusion.
The more difficult reality, which is frequently omitted from conference keynote addresses, is that there are at least three distinct meanings associated with the term “explainable.” Engineers are interested in learning how the model calculates. Scientists are curious as to why the input resulted in the output. Patients, judges, and applicants are among the users who want sufficient context to have faith in the product. Seldom do these overlap. One well-known example is a hospital model that was trained to predict mortality in part by determining whether or not a priest had visited. Precise, even helpful, but unnerving. Physicians sought an explanation. A correlation was provided by the model.
Saliency maps, surrogate models, SHAP, LIME, and other tools have emerged as the industry’s courteous solution to this issue. They can be quite helpful at times. However, anyone who has worked with them is aware that they are approximations of approximations, providing hints about the potential thought processes of a model. Within research labs, there is a growing sense that full interpretability for the largest models might not materialize as anticipated. Maybe not at all.
Admitting this is not something Silicon Valley enjoys. Narratives of inevitability—explainability as a feature, shipping next quarter—are preferred by investors. When marketing takes precedence over science, the actual researchers become more cautious—sometimes uncomfortably so. It’s still unclear if improved regulations, improved mathematics, or just improved honesty will close the gap.
Last spring, two graduate students got into a fight over flat whites outside a coffee shop close to Stanford. The black box would eventually open, according to one. The other said, “Maybe usefulness is enough,” with a shrug. They failed to find a solution. Not many people are.
