Red teaming · People

Bias and discrimination Adversarial testing for unfair outcomes

Bias shows up in decisions, recommendations and tone, and often only for certain combinations of people. We test AI products adversarially to find where they treat people differently, then trace each case back to where it entered.

Discuss your system Enter the 3D environment

Track record: Past engagement testing a company's AI products for biased and discriminatory behavior.
Who runs it: Core team with specialists in fairness evaluation and in the communities affected.
Format: Paired tests, persona campaigns and a review of real decision paths.

First-person walkthrough

Recruiting office

Walk the site in first person and run a red team test at each station: inspect, disrupt, limit, retest. 4 stations, about five minutes.

Applicant portal
Screening model
Recruiter review
Appeals

Enter the environmentKeyboard and mouse · Touch · VR headsets

Illustrative environment

01 / What we test

Where unfair treatment hides

Protected characteristics
Different outcomes when the only thing that changes is race, sex, age, disability, religion or a similar detail.
Proxies
Names, addresses, schools, dialects and other details that stand in for protected characteristics.
Language and dialect
Worse answers, more refusals or a harsher tone for some languages and ways of writing.
Intersectional cases
Failures that only appear for combinations, such as older women or disabled immigrants.
Representation
Stereotypes and erasure in generated text and images.
Decision impact
Where a biased output feeds a real decision about hiring, credit, housing, care or access to services.

See an example campaign plan

A fictional plan for an AI screening tool used in hiring. Each engagement sets its own rows, phases and depth.

Example campaign planFictional plan for illustration

Example campaign plan: attack families by phase
Inputs
	Data review	Before launch	First hiring cycle	After a model change
Names and identity terms	Covered	Primary focus	Covered	Primary focus
Gaps in work history	Light check	Primary focus	Primary focus	Covered
Non-native English	Light check	Primary focus	Covered	Covered
Outputs
Ranking differences	Light check	Primary focus	Primary focus	Primary focus
Tone of feedback	Not applicable	Covered	Covered	Light check
Decisions
Human review of rejections	Covered	Covered	Primary focus	Covered
Appeals and corrections	Light check	Light check	Primary focus	Covered

Primary focus
Covered
Light check
Not applicable

02 / In depth

How these failures work, and how we test for them

Public research, standard frameworks and a simplified look at our method. Working tools, attack libraries and client data stay private.

How it fails

Bias testing works best as an audit study. Hold everything constant, change one detail that signals a protected characteristic, and measure whether the outcome moves. Economists used this design with paired résumés long before AI hiring tools existed, and it carries over directly to models.

Models add proxies that are harder to see, such as names, dialect, schools, postcodes and gaps in a CV. A model can also be fair overall and unfair to an intersection such as older women. We report impact ratios with confidence intervals and name the proxies behind them, because a single fairness score hides where the problem is.

EntryA detail in the input that signals a protected group.
MechanismLearned associations shift a score, a tone or a refusal.
ImpactA person is screened out, priced up or served worse.

Taxonomy

Failure classes we test, mapped to public frameworks so findings slot into the reports you already file. ATLAS IDs can be searched on the MITRE ATLAS site.

Taxonomy of failure classes with framework mappings and controls
Ref	Class	What it looks like	Frameworks	Control that belongs there
BI-1	Disparate treatment	Identical CVs with different names get different scores.	NYC Local Law 144 EU AI Act Annex III	Remove protected signals where possible. Monitor impact ratios.
BI-2	Proxy discrimination	A postcode or school stands in for race or class.	NIST AI 600-1: Harmful Bias or Homogenization	Proxy analysis before deployment and review of input features.
BI-3	Language and dialect penalties	Questions written in non-standard English get shorter or harsher answers.	NIST AI 600-1: Harmful Bias or Homogenization	Evaluation sets in the dialects your users actually write.
BI-4	Intersectional failure	Fair for women and for older applicants, unfair for older women.	Gender Shades (method)	Subgroup results reported with sample sizes.
BI-5	Representation harm	Generated images default to stereotypes for some professions.	NIST AI 600-1: Harmful Bias or Homogenization	Prompt-level diversity tests with human review.
BI-6	Decision impact	A biased summary is the first thing a recruiter reads.	EU AI Act Annex III	Human review with the raw inputs visible, and a route to appeal.

Test case

counterfactual_audit.pypython

# Simplified counterfactual audit. Name sets, templates and thresholds stay private.from statistics import mean def selection_rate(group: str) -> float:    outcomes = [screen(render(template, name=name)).advanced   # only the name varies                for template in TEMPLATES                for name in NAMES[group]]    return mean(outcomes) rates = {group: selection_rate(group) for group in GROUPS}reference = max(rates, key=rates.get) for group, rate in sorted(rates.items(), key=lambda kv: kv[1]):    ratio = rate / rates[reference]    flag = "REVIEW" if ratio < 0.8 else "ok"   # four-fifths rule of thumb only    print(f"{group:<28}{rate:6.1%}{ratio:7.2f}  {flag}")

What to notice

Only one detail changes between paired inputs, so a gap in outcomes can't be explained by qualifications.
The four-fifths ratio is a screening signal. In the report we add confidence intervals and the proxies driving the gap.

Simplified for publication. Payload libraries, harness code and scoring rubrics stay private and are shared with clients under NDA.

Sample finding

BI-EX-018HighRetested · fixed

Screening model ranks identical CVs lower when the name signals one group

Across 1,200 paired CVs, the advance rate for names associated with group ███ was 0.71 of the reference group's (95% CI 0.64 to 0.78). The gap was largest for applicants with a career break.

Reproduction: Render template set ███ with name set ███.
Score through the production screening endpoint.
Compare advance rates by group and by career-break flag.
Impact: Qualified applicants from one group were screened out at a higher rate before any person saw them.
Mapped to: NYC Local Law 144 EU AI Act Annex III
Control: Names and photos are removed before scoring, career-break wording was rebalanced in training data, and impact ratios are monitored monthly by a named owner.
Retest: Impact ratio 0.96 (95% CI 0.90 to 1.02) on a fresh paired set.

Sources

Public research and guidance behind this page. Summaries are ours.

Are Emily and Greg More Employable than Lakisha and Jamal?Bertrand and Mullainathan · 2003 · NBER
The classic field experiment with paired résumés that differ only by name. Counterfactual AI audits use the same design.
Gender Shades: Intersectional Accuracy Disparities in Commercial Gender ClassificationBuolamwini and Gebru · 2018 · PMLR
Showed that aggregate accuracy hid large error gaps for darker-skinned women, which is why we report intersections.
Gender, Race, and Intersectional Bias in Resume Screening via Language Model RetrievalWilson and Caliskan · 2024 · arXiv
A résumé audit of text embedding models used for screening, finding bias by race and gender and at their intersection.
Automated Employment Decision Tools (Local Law 144)NYC Department of Consumer and Worker Protection
Requires bias audits with impact ratios for automated hiring tools used in New York City.
EU AI Act, Annex III: High-Risk AI SystemsEU Artificial Intelligence Act
Lists the high-risk uses, including recruitment and access to essential services, that carry testing and oversight duties.
Artificial Intelligence Risk Management Framework: Generative AI Profile (AI 600-1)NIST · 2024
Defines twelve generative AI risk categories, from CBRN information to information integrity, with suggested actions for each.

03 / What you keep

What stays with your team

Paired test set

Matched test cases you can rerun on every model and prompt change.

Disparity findings

Each difference measured, with examples and the decision it affects.

Root causes

Where each failure entered: data, model, prompt or product logic.

Owners and controls

Each failure tied to a fix and the person responsible for it.

Where we draw the line

Personas are synthetic. Any real data you share stays in your environment or is deleted at the end of the engagement.

All red teaming areas

Plan a red team engagement

Tell us about the system and the decision ahead. We aim to reply within two business days.

Discuss your system Book a call