Decoding Black Boxes
Explanation tooling for tabular models is mature and boring — SHAP still ships maintenance releases, and regulators already assume you use it. For language models in production, nobody explains the model. They constrain it, and I think that's the honest thing to build.
"How do we explain the model" now has two answers, and they've drifted a long way apart. If the model is gradient-boosted trees scoring loan applications, explanation is a solved engineering problem with maintained libraries and a compliance workflow bolted to it. If the model is an LLM in a pipeline, nobody explains it. They constrain it. The first version of this post treated both halves as one problem with different tooling. That was wrong, and it's the part worth rewriting.
The boring half is boring because it works
LIME landed at KDD 2016, SHAP at NIPS 2017, and a decade on they are still the workhorses for structured data. SHAP cut 0.51.0 on 4 March 2026 — NaN fixes in path-dependent TreeSHAP with small background sets, and other maintenance of the kind you want from something load-bearing. Nothing superseded them. The movement was underneath: TreeSHAP made exact Shapley values tractable on ensembles, which is why SHAP won the tabular space outright instead of remaining a nice idea with a factorial cost.
The regulation grew into the tooling rather than the other way round. CFPB Circular 2022-03, from May 2022, says a creditor can't excuse a vague adverse-action notice by pointing at model complexity: ECOA wants specific principal reasons whatever technology reached the decision. In practice, per-application attributions, ranked, mapped to reason codes, stored. In the EU, Annex III high-risk obligations moved from 2 August 2026 to 2 December 2027 under the Digital Omnibus agreement of 7 May 2026 — a 16-month deferral that buys documentation time and changes nothing about what the documentation contains.
Counterfactuals answer a better question for the applicant — what would have had to be different — and cost more to run, because the constraint set that keeps the search from returning "be six years younger" is domain knowledge someone writes down by hand.
The honest limit on all of it: an explanation is a second model's claim about the first, and it can be attacked. Slack et al., AIES 2020, built a scaffold that detects when it's being probed by LIME or SHAP's perturbation sampler and answers with an innocuous surrogate — a demonstrably racist classifier producing clean attributions on demand. Perturbation-based explainers query off-distribution by construction, and off-distribution is where a model can be taught to behave. Attributions from a model you don't control need checking like any other claim from an interested party.
What attention was never going to give you
Attention weights got treated as an explanation channel for years because they're right there and they visualise nicely. Jain and Wallace showed in 2019 that you can often find alternative attention distributions producing the same prediction, which breaks the claim that the original weights explain it. Wiegreffe and Pinter pushed back the same year and still landed on caution. A heatmap of where computation went is a different claim from why the answer came out as it did.
The research frontier is real, and it isn't a production channel
Mechanistic interpretability has produced genuine results and I won't be dismissive of them. Anthropic open-sourced its circuit-tracing tools on 29 May 2025, letting you build attribution graphs over open-weights models and step through them interactively; the demos run on Gemma-2-2b and Llama-3.2-1b. Anthropic's own framing is that the graphs reveal the internal steps only partially.
The counter-datum is more instructive. On 26 March 2025 the DeepMind interpretability team published negative results on sparse autoencoders and said they were deprioritising fundamental SAE research. On detecting harmful intent, with new jailbreaks as the out-of-distribution set, plain dense linear probes on raw activations were near-perfect while the SAE probes distinctly degraded — and probes trained on SAE reconstructions were also significantly worse, which suggests the autoencoder discards what the task needs. Their word for the field was "over invested".
That's a healthy field doing science. It is not a thing I can put in front of an auditor in 2026, on a model I access over an API, at a scale two orders of magnitude past Gemma-2-2b.
The half nobody explains
So here's my position, and I'll argue it rather than assert it.
The tempting answer for LLMs is to read the reasoning trace. It's natural language, it's free, and it looks like an explanation. Anthropic tested that in April 2025: slip a hint into a question, see whether the model admits using it. Claude 3.7 Sonnet mentioned the hint 25% of the time; DeepSeek R1, 39%. Both used hints far more often than they mentioned them, and faithfulness fell further on the concerning prompts — the ones where you'd most want to know. A trace that omits the operative cause three times in four is prose about the answer, generated by the process that generated the answer.
Given that, the thing you can build is a system whose behaviour is bounded even though the model's reasoning is opaque. Three parts, in the order I'd add them.
Ground the generation and gate it. The model answers from retrieved passages or refuses, and every citation is checked mechanically against the source text before the answer leaves the pipeline. Grounding alone isn't sufficient, and the numbers say so: the Stanford study of purpose-built legal RAG products, testing tools as of May 2024 and published in the Journal of Empirical Legal Studies in 2025, found hallucination rates between 17% and 33%, with one tool accurate on 42% of queries. Retrieval narrows the space the model can invent in. The check catches the remainder.
Make the downstream deterministic. Where the output drives an action, the model picks from an enumerated set and code does the rest — same input, same effect, testable with no model in the loop. Free text going straight into an effect is the design I'd argue hardest against, because there you inherit the model's whole output distribution as your failure surface.
Log the pipeline, not the intent. Retrieved document and revision, prompt hash, model version, the gate's verdict, the output, who saw it and when. Reconstructible after the fact by someone who wasn't there.
None of this explains the model, and I won't dress it up as though it does. It's a genuine loss: the tabular half of this work gives me a defensible per-decision reason and the language-model half doesn't, and I've stopped expecting the gap to close on tooling I can buy. What the constraints buy instead is a bounded system with a reconstructible record.
So I can hand over the passage, its revision, the model version and prompt hash, the deterministic step that consumed the output, and the name of the person who signed it off. I can't hand over why those tokens and not others. The tabular half has had an answer to that since 2017; the language-model half doesn't, and the tools that might get there are being demoed on a 2-billion-parameter open model.