Hiding someone's name doesn't hide their race from an AI

The intervention, first

That a ZIP code, a home address or an alma mater carries race is not a discovery. It is the premise of redlining, decades of housing and lending scholarship, and the reason US legal scholars named proxy discrimination as the distinctive risk of AI decision systems.^[7] Anyone who has thought about this for ten minutes expects the leak. The open question is what you can actually do about it.

So here is our answer before the setup.

An AI screening tool does not read a résumé as text. It converts it into a vector — a long list of numbers — and everything downstream, the ranking and the shortlisting, runs on that list. That list is what we measured.

Show the model two résumés that are identical word for word, differing in exactly one detail, and ask which applicant is Black. Scores run 0.5 to 1.0, where 0.5 means it is guessing at random and 1.0 means it is right every time:

The two proxies beat the name, on all four models.

This is a test of what screening vendors already do

Name-blind screening is not hypothetical. Redacting the candidate's name before the document reaches the model is standard practice — it is sold as a feature, it appears in procurement checklists, and several vendors strip the name from the text so it never reaches the embedding at all.

Each row above measures exactly that situation. The rows are run in isolation: in the address row, the only identifying detail on the résumé is Address: Chicago, IL 60619. No name anywhere. In the university row, the only detail is the education line.

So the address row is the name-blind pipeline. Strip the name completely, exactly as the vendors do, and the model still identifies race 49 times out of 50.

Redaction removes the 0.945 signal. It leaves the 0.984 and 0.983 signals untouched, because nobody redacts a home address — you need it — and nobody redacts a degree, because the degree is the point of the résumé.

The one field that gets removed is the weakest of the three.

The workflow, step by step

In production this runs as three steps:

1. Strip the name from the résumé.
2. Send the nameless résumé to the model, which scores it and ranks it against everyone else.
3. Put the name back, so the recruiter knows who they are calling.

Step 3 cannot change the outcome. The name was not there when the score was computed, so the score does not depend on it. Restoring it afterwards is a display choice, not a safeguard.

That leaves everything resting on step 2 — and step 2 is the first row of the table above. The model reads the nameless résumé, finds the address and the university, and identifies race about 49 times out of 50. The ranking it hands back is still sorted by the attribute the redaction was meant to remove.

And the name returns at the worst possible moment. The evidence that names change hiring outcomes is evidence about people. Bertrand and Mullainathan mailed résumés to human recruiters;^[3] every correspondence audit since, including the 83,000-application study of large US employers,^[2] has measured humans reading names. That is where the name effect is established.

This workflow removes the name from the machine — where the address and the university both outscore it — and gives it back to the human, at exactly the point where a person reads the shortlist and decides who gets a call.

It protects the stage where the name matters least, and restores the name for the stage where it matters most.

Why this matters for an audit. A pipeline built this way can be described, accurately, as name-blind, and still produce a race-correlated ranking. An audit that checks whether the name was removed will pass it. An audit that measures whether the attribute is still readable will not.

Can you take it out again?

Not with one edit, and the way this is usually tested will tell you that you can.

The standard way of checking cannot detect the failure. Fit an edit to remove a signal, then test it with a detector trained on that same signal, and it reads 0.500 — random guessing — every time. The arithmetic guarantees that result whether or not anything was removed.

This is a documented failure mode, not our observation. Gonen and Goldberg showed it in 2019 for the best-known debiasing method of the era: the gender direction was removed, the standard evaluation reported success, and gender remained fully recoverable from the same vectors by clustering — the bias had been hidden, not removed.^[16] Kumar, Tan and Sharma formalised why in 2022, in a paper whose title is the finding: probing classifiers are unreliable for concept removal and detection.^[14] And the strongest guarantee available, LEACE, is explicitly scoped — it provably blocks linear predictors of the concept as labelled, which says nothing about a different labelling of the same attribute.^[13]

Test it across signals instead and the picture changes. Fit the edit on the name, then ask whether the address still gives the applicant away: it does, at 0.99. Removing one channel removes one channel. The others are untouched.

That is not a quirk of our method. It is what you should expect from the history above: segregation ran through housing, education and association separately, so each leaves its own separate trace. Close one door in a house built with several.

One thing this experiment never did, and the distinction matters. We did not ask any model to make a hiring decision. There is no shortlist here, no ranking, no score, no yes or no. We measured what a résumé's vector encodes and what an edit removes from it. The occupation probe is a stand-in for "task-relevant content is intact" — it is not a measure of whether someone is qualified, and it is certainly not a hiring outcome. A system that can no longer read race from its representation can still produce discriminatory decisions for reasons this test does not cover. What we can say is narrower and still worth having: the protected attribute is no longer available in the representation for a downstream ranker to use, and the professional content is still there.

What we measure, and what we don't claim. Detection here is measured against a linear reader — the same family that production ranking, scoring and retrieval systems actually use. That is the class of system a candidate's application passes through. We report what those systems can read; we do not make claims about what a determined adversary could reconstruct with unlimited effort, and neither should anyone else.

The rest of this post is how the test was built, what leaked and by how much, and every number in a downloadable file.

Why the leak is worth quantifying anyway

Anonymising CVs — stripping the candidate's name before anyone reads the application — is one of the most widely adopted remedies for hiring bias, and a standard line item in both procurement checklists and screening products.

The problem it is meant to solve is well established in the US and has not improved. A meta-analysis of every US field experiment on hiring discrimination since 1989 found no decline in discrimination against Black applicants over 25 years.^[1] The largest US correspondence audit to date — roughly 83,000 fictitious applications sent to 108 of the largest US employers — found applications with distinctively Black names received materially fewer callbacks, and that the effect was concentrated in a subset of firms rather than spread evenly.^[2] This is the same design Bertrand and Mullainathan introduced in 2004.^[3]

But removing the name has already been shown to fail, and sometimes to backfire. A randomised French experiment found anonymous résumés reduced interview rates for minority candidates,^[4] a Swedish trial found partial effects at best,^[5] and the Australian Public Service's randomised trial found de-identification did not deliver the expected benefit.^[6]

US legal scholarship names the reason. Proxy discrimination — where a facially neutral variable stands in for a protected attribute — has been argued to be the central and distinctive risk of AI and big-data decision systems,^[7] and the specific strategy of removing protected inputs has been criticised as an "input fallacy": restricting what the model is told does not restrict what it can infer.^[8]

And the harm has now been measured at scale in deployed systems. A Stanford study of 4 million applications from 3.4 million people across 1,700 postings at 150 employers — all scored by a single vendor's AI hiring tool — found that 26% of Black applicants applied to positions where the system disadvantaged their group under the EEOC four-fifths rule, and that roughly 40,000 more applications would have advanced at parity. It also documented "algorithmic monoculture": because one vendor scores everyone, 10% of applicants who submit four applications are rejected from all four.^[26]

We want to be precise about how our work relates to theirs, because they are not the same thing and theirs is bigger.

Their evidence is far stronger than ours on the question they asked. Four million real applications at 150 real employers is external validity we cannot approach with 500 synthetic pairs on four open models. Their monoculture finding — the same vendor rejecting the same person everywhere — is something our design structurally cannot produce.

But an observational outcome study cannot say which feature carried the disparity, and it proposes no remedy. That article describes no mitigation, and by design it could not: you cannot isolate a cause from outcomes alone when every applicant differs in a hundred ways at once. An employer who reads it learns they have a problem, not what to change.

That is the gap this post fills. A matched-pair design cannot tell you how much harm occurs in the world — but because the two résumés are byte-identical, it can isolate exactly which line carries the signal, and it can measure what is left after you intervene. Which proxy, how much, and what remains afterwards.

Measured on résumés that are byte-for-byte identical except for one injected detail:

(0.5 = the model cannot tell at all. 1.0 = perfect. Source: resume_full_ALL.csv.)

Both proxies beat the name, on all four models — so deleting the name moves you from 0.945 to 0.984, in the wrong direction. That is the whole finding: the thing every blind-hiring policy removes is the weakest of the three signals it leaves behind.

Neither proxy asks you to trust our judgement. HBCU status is fixed in federal statute and the US Department of Education publishes the official register^[25] — every school in our test appears on it. The racial composition of all twelve ZIP areas is published in full below, retrieved from the 2020 Census.^[12] Both premises are checkable against government sources.

How the test was built

This is a correspondence audit — the design Bertrand and Mullainathan used in their 2004 study, where researchers mailed otherwise-identical résumés differing only in the name.^[3] We run the same logic inside the model rather than through the mail.

The résumés are real. We use biographies from the public bias_in_bios dataset,^[9] each labelled with the person's profession. That profession label is our control signal: it is professional content the edit must not damage. It stands in for "the useful information is still there" — it is not a measure of whether the person is qualified, and no model in this experiment is asked to judge that.

Each pair differs by exactly one thing. For every biography we produce two versions with identical body text, character for character. Only the injected line differs:

Because the qualifications are literally the same sentences in both arms, any difference the model detects is the injected attribute.

Where the names come from. Gender first names are the most common US names for the 1980s birth cohort from the Social Security Administration,^[10] so they describe people of working age. Surnames come from the 2010 US Census surname file,^[11] loaded directly by the script — and the same surname appears in both arms of a race pair, so the first name is the only race signal. Race first names are taken from Bertrand and Mullainathan's published Table 1.^[3]

Where the education proxy comes from. Eight Historically Black Colleges and Universities — Howard, Morehouse, Spelman, Hampton, Florida A&M, North Carolina A&T, Tuskegee, and Xavier of Louisiana — against eight deliberately generic public schools ("State University", "Regional State University" and similar).

Where the address proxy comes from, and what each ZIP actually is. Twelve ZIP Code Tabulation Areas. We retrieved the racial composition of every one of them from the 2020 Decennial Census, Demographic and Housing Characteristics file, table P8, queried per-ZCTA through the Census Bureau's API.^[12] These are the figures, not estimates or characterisations:

Nothing sits in the middle. Six ZCTAs fall between 81.0% and 96.0% Black; six fall between 0.4% and 2.1%. There is no area in this test anywhere near the national figure. Kenilworth, Illinois has ten Black residents out of 2,514. Jackson 39213 is 96.0% Black. That gap is the contrast the address axis actually tests — and it is also the reason the axis cannot show a dose-response, since we sampled only the two ends.

Every figure above is downloadable at zcta_race_2020.csv, and the retrieval script is scripts/fetch_zcta_race.py, so the query can be re-run against the Census API rather than taken on our word.

The address result, and the three things it still cannot tell you

The home address axis scores 0.984, just ahead of the education line. Its premise is now measured rather than assumed — every ZCTA in the table above was retrieved from the 2020 Census, and all twelve confirm the composition the design relied on. But three limits remain, and none of them are fixed by having the census figures.

1. The ZIP is confounded with the place name. The injected line reads Kenilworth, IL 60043, not 60043. A model that has never learned a single ZIP code, but knows Kenilworth is an affluent North Shore suburb and Jackson is not, scores identically. So the honest unit here is the address line, not the ZIP code. Isolating the digits requires re-running with bare ZIPs.

2. Six addresses per arm, not six hundred. Across 500 pairs the same twelve strings recur. Part of what the probe learns is to recognise a small fixed set. That does not make the leakage unreal — a real screener also meets a bounded set of place names — but the effect size is not a statement about US ZIP codes in general.

3. Picked from the extremes, so this cannot show a dose-response. A 96%-Black ZCTA against a 0.4%-Black one is the easiest possible contrast. The stronger experiment samples across the full composition range and asks whether readability scales with the percentage. We have the retrieval script now, so that version is cheap to run — it just has not been run.

What the axis does establish, cleanly: given a résumé identical in every respect except a home address in a 96%-Black versus a 0.4%-Black area, four different models tell them apart essentially perfectly, and more reliably than they tell apart a distinctively Black name from a distinctively White one.

The two results that carry the argument

1. The ordering is consistent. On all four models — from three different labs — the education line scores above the distinctively-raced name. Not an averaging artifact: 4 out of 4. The address axis shows the same ordering, with the caveats noted above.

One limit that applies to both proxies and belongs in the open: across 500 pairs there are eight distinct schools per arm and six distinct addresses, cycled. Part of what the probe learns is to recognise a small set of strings. That does not make the leakage unreal — a production screener meets the same finite set of university names — but it does mean the effect size should not be read as a general statement about all HBCUs or all ZIP codes.

2. The signal can be removed, and qualifications survive. The edit is LEACE — least-squares concept erasure, published by Belrose et al. and available under an MIT licence.^[13] We are naming it plainly because there is nothing proprietary here: the method is public, the reference implementation is free, and anyone can apply it. What is ours is the measurement around it, not the projection.

Concretely, for a binary attribute the procedure is: centre and whiten the embeddings, take the difference of class means in that whitened space as the concept direction, project it out, then un-whiten. That is the rank-1 LEACE solution, not a variant of it — whitening is the mechanism LEACE prescribes, not something we added. Two implementation details a replicator needs: we compute it directly rather than calling the reference concept-erasure package, and we add a small ridge term (1e-3) to the covariance before inverting, with eigenvalues floored at 1e-6.

We then train a brand-new logistic-regression probe from scratch on the edited embeddings. Tested on the same signal the edit was fitted on, it lands at chance — but that is the circular check described above and it is not evidence. Tested on a signal the edit never saw, the attribute is still readable at 0.99. What the edit does not do is damage the résumé: the model's ability to predict the person's profession changes by a mean of 0.002, worst case 0.008.

The vector loses what it encoded about who the candidate is and keeps what it encoded about their professional life. What a downstream ranker then does with that vector is a separate question, and one this experiment does not answer.

What "erased" is allowed to mean

Here is the boundary of the result above, stated exactly.

The post-erasure probe is linear. It is logistic regression — the same family as the detector that read the attribute in the first place, and the family that production ranking, scoring and retrieval systems actually use. Against that class of reader, on this data, the attribute is gone: chance, not merely reduced.

We did not run a non-linear probe after erasure on this data. The multi-layer probe in this experiment runs before the edit, never after. So this run establishes that the attribute is linearly unreadable; it does not establish that it is unrecoverable. Those are different claims and only the first is measured here.

We are drawing that line sharply because the broader literature says the gap is real. Probing-based removal has been shown not to guarantee a concept is unrecoverable,^[14] and adversarially-optimised linear erasure still leaves non-linear recoverability open.^[15] In our own work on other datasets, determined non-linear readers recover a meaningful fraction after linear erasure — we are deliberately not quoting those figures here, because they were measured on different data with different concepts and carrying them across would be exactly the kind of borrowed number this post exists to argue against. The résumé-specific version of that measurement is one function call, and it is queued.

The claim we will sign is therefore precise: provably unusable by a standard linear reader, with the non-linear residual measured and reported per dataset rather than assumed. Any vendor claiming a protected attribute is simply "removed" — full stop, no reader class named — is overselling.

One engineering constraint that follows: if edited vectors are re-normalised to unit length — which most embedding pipelines do by default — part of the guarantee leaks back, because the vector's length carries what was removed from its direction. Rankings are unaffected, so a quality test will not catch it.

Per-model results

Read down the last two columns: on all four models, from three different labs, the address and the education line score above the name. The mean of each column is the headline figure — 0.984, 0.983, 0.979, 0.945.

How this maps to what regulators are asking

United States, and already in force:

• NYC Local Law 144 of 2021, effective July 2023, requires an annual independent bias audit of automated employment decision tools used for hiring or promotion in New York City, with results published.^[17] This test is the shape of that audit — except that LL144's required metrics are computed on outcomes, whereas what we measure is the readability of the attribute in the representation. Both matter, and an audit that reports only the first can miss a proxy the model is plainly using.
• EEOC technical assistance (May 2023) confirms that Title VII's disparate-impact framework applies to algorithmic selection procedures, and that an employer is generally responsible for a tool even when a vendor built it.^[18] That is the point our HBCU and address results bear on directly: neither is a protected attribute, both function as one.
• NY DFS Insurance Circular Letter No. 7 (2024) directs insurers to test for proxy discrimination on a cadence, requires independent review, and states a firm may not rely solely on a vendor's assertion of non-discrimination.^[19]
• Illinois has regulated AI video interview analysis since 2020,^[20] and from 1 January 2026 a second Illinois statute prohibits employment AI that produces a discriminatory effect on a protected class — and expressly bars using ZIP code as a proxy for a protected class.^[21] That is the exact practice measured in this post: our address axis reads race at 0.984, higher than the name does. Colorado enacted a broad AI-discrimination statute in 2024 covering employment decisions — though its status has since changed and anyone relying on it should check the current position rather than cite it as settled.^[22]

International and framework:

• EU AI Act (Regulation (EU) 2024/1689) places employment-related screening in its high-risk category, with obligations covering bias examination and technical documentation.^[23]
• NIST AI RMF 1.0 — "Measure" and "Manage" become concrete: measure whether the attribute is readable, manage it with a documented edit, report the residual.^[24]

Boundaries

• A controlled simulation, not an applicant-tracking system. Re-run it on your own pipeline before relying on it.
• No model in this experiment made a hiring decision. There is no ranking, shortlist, score or yes/no anywhere in it. We measured what a résumé's vector encodes and what an edit removes. A system whose representation no longer carries race can still produce discriminatory outcomes through its decision rule, its training data, or its thresholds — none of which this test touches.
• The occupation probe is a control, not a qualification measure. It shows professional content survived the edit. It does not show that a good candidate still ranks highly, because nothing here ranks anyone.
• The address result is confounded with city name, and the ZIP set is curated from the extremes, not sampled across the composition range.
• Erasure is verified against a linear reader only. The post-erasure probe on this data is logistic regression; no non-linear probe was run after the edit. "Linearly unreadable" is what we measured. "Unrecoverable" is not.
• The whitening statistics are fitted on the full embedding set. The erasure direction is fitted on the training split alone, which is the part that determines the edit — but the whitening mean and covariance see the held-out split too. We are fixing this to train-only and will re-publish if any number moves.
• Four axes only. Employment gaps, sports, memberships, finer geography — untested. The proxies you did not think to check are the ones to worry about.

Reproduce

• Full results, all 16 rows: resume_full_ALL.csv
• Wider operator battery: audit_summary_2026-07.csv
• Per-backbone detail: gemma-2-9b · phi-3.5-mini · mistral-7b · qwen2.5-1.5b · qwen2.5-7b

Every name list, school list and ZIP area is printed above, so the test can be rebuilt without our code. Probes train on a bio-level held-out split — no biography appears in both train and test — and the detector used to check erasure is trained after the edit, on data the eraser never saw.

If your team wants this battery run against your own screening pipeline, talk to us.

References

US evidence on hiring discrimination

1. Quillian, L., Pager, D., Hexel, O., & Midtbøen, A. H. (2017). "Meta-analysis of field experiments shows no change in racial discrimination in hiring over time." Proceedings of the National Academy of Sciences, 114(41), 10870–10875. ↩
2. Kline, P., Rose, E. K., & Walters, C. R. "Systemic Discrimination Among Large U.S. Employers." Quarterly Journal of Economics (NBER Working Paper 29053, 2021). Approximately 83,000 fictitious applications to 108 of the largest US employers. ↩
3. Bertrand, M., & Mullainathan, S. (2004). "Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination." American Economic Review, 94(4), 991–1013. ↩

Evidence that anonymising applications does not fix it

4. Behaghel, L., Crépon, B., & Le Barbanchon, T. (2015). "Unintended Effects of Anonymous Résumés." American Economic Journal: Applied Economics, 7(3), 1–27. ↩
5. Åslund, O., & Nordström Skans, O. (2012). "Do Anonymous Job Application Procedures Level the Playing Field?" Industrial and Labor Relations Review, 65(1), 82–107. ↩
6. Behavioural Economics Team of the Australian Government (BETA), Going Blind to See More Clearly: Unconscious Bias in Australian Public Service Shortlisting Processes, 2017. ↩

US legal scholarship on proxy discrimination

7. Prince, A. E. R., & Schwarcz, D. (2020). "Proxy Discrimination in the Age of Artificial Intelligence and Big Data." Iowa Law Review, 105, 1257. ↩
8. Gillis, T. B. (2022). "The Input Fallacy." Minnesota Law Review, 106, 1175. ↩

Data sources used in this experiment

9. De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. (2019). "Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting." Proceedings of FAT* 2019. ↩
10. US Social Security Administration, Popular Baby Names by birth cohort. ssa.gov/oact/babynames ↩
11. US Census Bureau, Frequently Occurring Surnames from the 2010 Census. census.gov ↩
12. US Census Bureau, 2020 Decennial Census, Demographic and Housing Characteristics (DHC) file, Table P8 (Race), by ZIP Code Tabulation Area, retrieved via the Census Data API (api.census.gov/data/2020/dec/dhc). Retrieved figures for all twelve ZCTAs: zcta_race_2020.csv; retrieval script scripts/fetch_zcta_race.py. ↩

Methods

13. Belrose, N., Schneider-Joseph, D., Ravfogel, S., Cotterell, R., Raff, E., & Biderman, S. (2023). "LEACE: Perfect Linear Concept Erasure in Closed Form." NeurIPS 2023. ↩
14. Kumar, A., Tan, C., & Sharma, A. (2022). "Probing Classifiers are Unreliable for Concept Removal and Detection." NeurIPS 2022. ↩
15. Ravfogel, S., Twiton, M., Goldberg, Y., & Cotterell, R. (2022). "Linear Adversarial Concept Erasure." ICML 2022. ↩
16. Gonen, H., & Goldberg, Y. (2019). "Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them." NAACL-HLT 2019. ↩

Regulation and guidance

17. New York City Local Law 144 of 2021, Automated Employment Decision Tools, enforcement effective 5 July 2023; requires an annual independent bias audit with published results. ↩
18. US Equal Employment Opportunity Commission, technical assistance document, Assessing Adverse Impact in Software, Algorithms, and Artificial Intelligence Used in Employment Selection Procedures Under Title VII, May 2023. ↩
19. New York State Department of Financial Services, Insurance Circular Letter No. 7 (2024), Use of Artificial Intelligence Systems and External Consumer Data and Information Sources in Insurance Underwriting and Pricing. ↩
20. Illinois Artificial Intelligence Video Interview Act, 820 ILCS 42, effective 2020. ↩
21. Illinois HB 3773 (2024), amending the Illinois Human Rights Act, effective 1 January 2026: prohibits an employer from using artificial intelligence that has the effect of discriminating on the basis of a protected class in recruitment, hiring, promotion and other employment decisions, and prohibits the use of ZIP code as a proxy for a protected class. ↩
22. Colorado SB 24-205 (2024), consumer protections for artificial intelligence covering consequential decisions including employment. Status has changed since enactment — verify the current position before relying on it. ↩
23. Regulation (EU) 2024/1689 (Artificial Intelligence Act), high-risk classification covering employment and worker-management systems. ↩
24. National Institute of Standards and Technology, AI Risk Management Framework 1.0 (NIST AI 100-1), January 2023. ↩

Definitions

25. Historically Black Colleges and Universities are defined in the Higher Education Act of 1965, Title III, Part B; the official list of eligible institutions is maintained and published by the US Department of Education. All eight schools used in the education axis appear on it. ↩

Related work

26. Bommasani, R., Bana, S. H., Creel, K. A., Jurafsky, D., & Liang, P. Study of algorithmic hiring across 4 million applications from 3.4 million applicants, 1,700 postings and 150 employers processed by a single vendor. Summarised by Stanford HAI, "AI Hiring Tools Can Yield Racial Bias and Systemic Rejection"; paper at algorithmichiring.github.io. ↩

---

Conducted July 2026 on frozen open-weight models' hidden states — four models, four axes, 500 matched pairs per axis, mean-pooled representations at 60% depth, with bootstrap confidence intervals on detection scores. Concept erasure and linear probing are published science, credited above; our contribution is the measurement, run honestly and reported with its limits.