Navigating Explainability

I operate an enterprise AI gateway: a proxy that every internal model call goes through, doing routing, key management, usage accounting and audit. When someone arrives at my desk with why did the model do that, I usually can't tell them. What I can tell them is which model version served the request, which key issued it, what went in, what came out, what it cost, and when. Thinner than an explanation, and the only artefact I've ever managed to produce on demand for someone who needed it that afternoon.

Three questions wearing one word

Attribution — which inputs moved this prediction. SHAP, LIME, permutation importance, the saliency-map family.

Documentation — what this model is, what it was trained and evaluated on, where it's known to fail. Model cards (Mitchell et al., FAT* 2019) are still the canonical form, and at seven years old still the most useful thing most teams could write this quarter and don't.

Record — what happened on this specific request, at 14:32, under this configuration, for this user.

Nearly all the interesting writing is about the first. Nearly every question I field in production is the third, and most of the regulatory text I've had to read is the second and third.

The attribution tooling, honestly

I'm not down on SHAP and LIME. For tabular models with a contestable decision behind them — a declined application, a flagged transaction — they're the right tool. Two things get skipped in the summaries.

Cost is inherent to the method, not an implementation detail you can optimise away. Exact Shapley values mean evaluating the model over every subset of features, so the work grows as 2^M; KernelSHAP exists to dodge that, approximating at roughly O(M²), with estimate quality scaling in how many samples you'll pay for. In development that's a slow notebook cell. In production, if you've promised an explanation next to every decision, it's a second inference budget with its own latency profile and its own failure modes.

The other is that the explanation is itself an artefact you have to trust. Slack et al., AIES 2020 built a scaffold that detects the perturbed inputs LIME and SHAP feed a model while sampling, and answers those innocuously while the real classifier stays biased on real traffic. The explainer reports a clean model. The model isn't clean. That result is six years old and I still see explainer output treated as ground truth rather than as another system output with a threat model.

Rudin's 2019 argument went further: for high-stakes decisions, use models interpretable from the start instead. She's right about the domains she writes about, and it has almost no purchase on what arrives at my gateway, which is large, opaque, and someone else's.

The generative case is worse

For an LLM there's a tempting shortcut: ask it. The reasoning trace is right there, in fluent English, free.

Anthropic planted hints in prompts, kept the cases where the answer changed as a result, and checked whether the chain of thought admitted using the hint. Claude 3.7 Sonnet mentioned it 25% of the time; DeepSeek R1, 39% (May 2025). Most of the time an influence was demonstrably present, the stated reasoning didn't contain it.

A plausible explanation uncorrelated with the cause is worse than none, because someone will paste it into an incident report.

What the regulators asked for

I expected the EU AI Act to demand explanations. It mostly demands paperwork and logs.

Article 12 requires high-risk systems to technically allow automatic recording of events over their lifetime, sufficient to identify risk situations and support post-market monitoring. Article 13 is instructions for deployers: concise, complete, accurate, clear. Article 26(6) puts a floor under retention — deployers keep the logs in their control for a period appropriate to purpose, and at least six months.

Recording, monitoring, instructions. Not one saliency map in the requirement.

The dates moved, and that's the instructive part. General-purpose model obligations started 2 August 2025 and the Article 50 transparency deadline holds at 2 August 2026. But the high-risk obligations due that same day were deferred by the Digital Omnibus — Annex III standalone systems to 2 December 2027, Annex I embedded systems to 2 August 2028 — agreed in early May 2026, largely because national authorities and harmonised standards weren't ready.

Eighteen extra months sounds like relief. Not for the record-keeping, because logs are the one compliance artefact you can't produce retroactively: if the system runs in 2026 and the obligation bites in 2027, the 2026 traffic is either recorded or gone. The NIST AI RMF (AI 100-1, 2023) arrives at the same place voluntarily — GOVERN, MAP, MEASURE, MANAGE, leaning throughout on systematic documentation.

The boring layer, in build order

A request record with a pinned model version. Our first cut logged the model alias, the friendly name teams call. Aliases move. When the target behind one changed, a fortnight of records went unattributable: I had the request and the response and no defensible answer on which weights produced them. Log the resolved provider model identifier, with the alias as a separate field.

Identity on every call. Which key, which team, which human behind the key. Attribution in the accounting sense, not the Shapley sense, and it answers more real questions than any explainer I've deployed.

Token counts and cost. Partly because it's the number that gets management attention, which is what pays for the rest of this.

A retention decision made before the logs exist. Prompt and response capture is the most useful field in the record and the most dangerous thing in the estate. Six months is the legal minimum; pick your own number and be ready to defend it to whichever of security or legal asks first.

Only then would I attach attribution tooling, and only where the model is tabular and the decision is contestable by whoever it landed on.

For a vendor-neutral record shape, the OpenTelemetry GenAI semantic conventions are the obvious candidate, worth adopting now with your eyes open. The v1.42.0 core release on 12 June 2026 moved the gen_ai.* attributes into a dedicated repository, which is not a graduation to stable: as of mid-July 2026 nothing GenAI-specific there is marked Stable, and there's no versioned release with a finalised schema URL to pin against. Note in your own docs which revision your emitters were written against.

What it doesn't do

None of this explains anything. A complete request record tells you what went in and what came out, is silent on why, and for a generative model I have nothing I'd put in front of an auditor as a causal account. It's necessary and nowhere near sufficient, and the second half of that belongs on the slide too.

What it does do is make answerable the questions people turn up with: which version served this, did routing change between the good output and the bad one, who spent that, show me every call this key made in March. Those I can answer in minutes. The retention floor sits at six months, and the logs are keyed on the resolved model identifier.