Start here
AI Bias in plain language
Bias can arise before a model is trained: some users may be missing from the data, a label may reflect an inconsistent process, or the chosen target may poorly represent the real need. Deployment choices can then amplify these problems.
An observed difference is a signal to investigate, not a complete explanation of its cause. Compare relevant error types, sample sizes and operating conditions before deciding what should change.
How it works
Identify who and what is represented
List relevant user groups and conditions. Check whether the evaluation covers the people and inputs the system will actually serve.
Compare meaningful errors
Measure the failures that matter for the task, alongside counts and uncertainty. Overall performance can conceal a weak subgroup.
Investigate and retest
Examine labels, inputs, thresholds and the surrounding process. Test any change across groups rather than fixing one score in isolation.
Illustrative example · Fictional data
When an average hides a routing gap
- Input
- A fictional support router misroutes 10 of 100 messages in language A and 10 of 20 messages in language B.
- Process
- Calculate separate error rates: A is 10%; B is 50%. The combined rate is 20/120, approximately 16.7%.
- Expected output
- A clear evaluation gap requiring investigation into coverage, labels and language handling.
- How to check it
- The smaller sample for B matters. This exercise does not prove a particular cause or establish a discrimination finding; it shows why the combined rate alone is insufficient.
Where it helps
- Checking whether service quality differs across relevant user groups.
- Reviewing data collection and labelling choices.
- Testing whether an update improves one group while worsening another.
Bias, random error and fairness
- Random error
- Individual mistakes without an established systematic pattern.
- Bias
- A systematic skew that can arise from data, design or use.
- Fairness
- A context-dependent assessment of treatment and outcomes; it is not settled by one universal metric.
Historical, sampling, measurement and deployment bias
Historical bias occurs when recorded outcomes reflect past inequities. Sampling bias occurs when the collected population does not represent the people who will use or be affected by the system. Measurement bias appears when a proxy or label does not measure the intended concept equally well across groups—for example, past hiring decisions used as a label for candidate suitability.
Bias can also arise from the objective, threshold or deployment process. A model optimised for average accuracy can perform poorly for a smaller group. Removing a protected attribute does not necessarily remove information correlated with it. Inspect labels, missing records, proxies and the surrounding decision process, not only the model.
Choose a fairness question before choosing a metric
Demographic parity compares positive-selection rates. Equal opportunity compares true-positive rates among people with a positive ground-truth outcome. Equalized odds also considers false-positive rates. Calibration asks whether a score has a similar observed meaning across groups. These are different questions and can conflict under some conditions; there is no single score that certifies a system as fair.
For an illustrative screening model, compare qualified candidates who were advanced and qualified candidates who were missed in each group. Report sample sizes and label limitations alongside rates. Agree which error causes harm, how uncertainty is handled and who can challenge the outcome before setting a threshold.
Mitigation before, during and after training
Pre-processing can improve representation, repair labels or reweight examples. In-processing can change the objective or introduce constraints. Post-processing can change decision rules or thresholds, subject to the use case and applicable requirements. Re-evaluate accuracy, group errors and operational consequences after any intervention.
Bias concerns systematic patterns of harm or performance differences. Hallucination concerns unsupported generated content. A statement can be factually correct but unfairly selected or framed; it can also be fabricated without a group-based pattern. The two problems need distinct tests and may coexist.
Related reading and learning pathways
Common pitfalls
- Removing a sensitive attribute does not remove correlated proxy information.
- Averages can hide gaps, while tiny subgroup samples can produce unstable rates.
- Equalising one metric may create other trade-offs that need an explicit decision.
Put the idea to work
Try it yourself
Recalculate the routing example if language B has 4 errors out of 20. Compare the group rates and overall rate, then list what evidence would be needed to explain the difference.
Common questions
Is AI bias only a training-data problem?
No. Objectives, labels, evaluation, thresholds and the way outputs are acted on also matter. For example, a threshold chosen for one region may disadvantage another, or staff may trust outputs more for some users. Review the full decision process, not only the model.
Can a high-accuracy model still be biased?
Yes. A strong average can coexist with poor performance for a smaller group or a particular input condition. Break results down by relevant groups, such as region, language or customer segment, and compare error rates to find gaps that the overall figure hides.
Can one test prove a system is bias-free?
No. Testing has a scope. Record what was measured, which populations were covered and what remains uncertain. Bias testing should be repeated after model updates, data changes and expansion to new users, because new conditions can create gaps that earlier tests did not cover.
Your next step
Continue learning
Connected concepts
Sources and further reading
Examples on this page are teaching exercises using fictional information. They are not client case studies or measured outcomes.