Start here
Explainable AI (XAI) in plain language
A developer may need to know which input features affect a prediction. A user may need to know what information was used and how to challenge an error. Those are different explanation needs.
Some models have logic that can be inspected directly. Post-hoc methods analyse a model after training, often with approximations. A feature-attribution result describes a model relationship under particular assumptions; it does not automatically identify a real-world cause.
How it works
Choose the explanation question
Decide whether the reader needs an overview of behaviour or an account of one particular prediction.
Select and test the method
Use a method suited to the model and input. Check its assumptions and whether small input changes cause unstable explanations.
Provide a useful next action
Show the relevant evidence and limitations. Where the output affects a decision, explain how incorrect input can be corrected or reviewed.
Illustrative example · Fictional data
Explaining a fictional queue rule
- Input
- A transparent teaching rule flags a support ticket when waiting time is over 48 hours and the ticket is unresolved. Ticket A has waited 60 hours and remains unresolved.
- Process
- Inspect both conditions: 60 exceeds 48, and unresolved is true. State both reasons, not just the waiting time.
- Expected output
- “Flagged because this unresolved ticket has waited more than 48 hours.”
- How to check it
- A resolved ticket at 60 hours should not be flagged. This exact rule explanation is a baseline; a post-hoc explanation of a complex model may be approximate rather than exact.
Where it helps
- Debugging unexpected model behaviour.
- Helping reviewers inspect the factors associated with an output.
- Communicating limitations and correction paths to affected users.
Local, global and causal explanations
- Local
- Describes one prediction or a small neighbourhood of inputs.
- Global
- Describes broader patterns in the model’s behaviour.
- Causal
- Addresses what changes would cause an outcome. This requires more than ordinary feature importance.
SHAP, LIME and counterfactual explanations
SHAP attributes a model output to features relative to a chosen reference or background. The meaning depends on the explainer, model and treatment of feature dependence. A large attribution means a feature influenced that output under those assumptions; it does not prove a real-world causal relationship.
LIME fits an interpretable approximation around a particular prediction by perturbing inputs and observing outputs. Its explanation is local: changing the neighbourhood, perturbations or random seed may change the explanation. Check that the approximation is faithful where you intend to use it.
A counterfactual asks what feasible input change would alter the prediction. Actionability matters: changing an immutable characteristic or an impossible combination of features is not useful recourse. Constraints and domain review are part of the explanation design.
Worked interpretation: a fictional support-priority model
Suppose a model scores a ticket as high priority. A local attribution identifies outage wording and the affected-user count as influential relative to its baseline. LIME might approximate the same prediction with a short local rule. A counterfactual might show that a smaller affected-user count changes the score while the ticket text stays fixed.
These outputs describe model behaviour, not whether the customer truly deserves priority. Check the raw ticket, the handling of missing counts and whether similar tickets get stable explanations. Keep the existing global/local/causal distinction: explanations can support review without certifying correctness, fairness or legal compliance.
Related reading and learning pathways
Common pitfalls
- A generated rationale can sound convincing without reflecting the model’s computation.
- An understandable model can still be inaccurate or unfair.
- A global importance ranking does not explain every individual result.
Put the idea to work
Try it yourself
Apply the queue rule to four tickets covering every combination of resolved/unresolved and below/above 48 hours. Write the explanation for each and test the boundary at exactly 48 hours.
Common questions
Is explainability the same as accuracy?
No. A model can be easy to explain and frequently wrong. Evaluate explanation quality and task performance separately. Simple models such as decision trees are easier to interpret but may be less accurate on complex data. Choose based on the stakes, the audience and regulatory needs.
Does feature importance prove causation?
No. Importance reflects a model’s use of information under the chosen method, not necessarily the cause of the real-world outcome. Two correlated features can share importance unpredictably. To understand cause and effect, use experiments, domain expertise or causal analysis alongside model explanations.
Can I ask an LLM to explain itself?
You can ask for a rationale, but treat it as generated content. Validate claims against observable inputs, tool results and independent checks rather than assuming access to its internal computation. Self-reported reasons can sound convincing yet differ from what actually drove the output.
Your next step
Continue learning
Connected concepts
Sources and further reading
Examples on this page are teaching exercises using fictional information. They are not client case studies or measured outcomes.