inside our bias audit: the rubric, the data, the changes
every quarter we run a bias audit on the dreamclerk interview. here is the rubric, the per-group pass-rates, and the three rubric changes we made after cohort 1.
we run a bias audit on the dreamclerk interview every quarter. the audit covers three things: (1) per-group pass-rate, (2) inter-rater reliability between the human reviewer and the AI reviewer, and (3) a regression of pass-rate on the four rubric dimensions, to see if any single dimension is doing the heavy lifting.
we publish the data because the alternative is that we are asking applicants to trust us. we would rather not be trusted; we would rather be checked.
the rubric
the rubric has 4 dimensions, each scored 1–4. the total is 4–16. to pass, you need ≥11 and no dimension <2.
- specificity — does the answer name a specific decision, not a generic principle? "i cached the user object in redis" beats "i used caching."
- honesty about the cost — does the answer acknowledge the cost of the wrong decision? "i should have benchmarked" beats "i should have thought more about it."
- revisit — has the answer been revisited since the original decision? "i would now use a different sharding strategy because the access pattern changed" beats "i would now write it better."
- transfer — does the answer say what the next decision will be, and why? "the next time i see this pattern, i will…" beats "i learned from it."
these four dimensions are not orthogonal. they correlate. a strong answer scores 3 or 4 on all four. a weak answer scores 1 or 2 on at least two.
the data
cohort 1 (n=212, march 2025):
| group | pass-rate | |---|---| | overall | 31.6% | | female | 28.9% | | male | 32.7% | | tier-1 college | 33.0% | | tier-2/3 college | 30.1% | | self-taught | 28.0% | | cs major | 32.4% | | non-cs major | 29.7% |
gender gap: 3.1 percentage points (within noise). college-tier gap: 2.9 pp (within noise). cs-vs-non-cs gap: 2.7 pp (within noise).
the noise ceiling on this sample size is roughly ±4 pp. so the gender gap is real but small. the college-tier gap is at the edge of noise. the cs-vs-non-cs gap is below noise.
cohort 2 (n=287, may 2025):
| group | pass-rate | |---|---| | overall | 33.4% | | female | 32.0% | | male | 33.8% | | tier-1 college | 33.9% | | tier-2/3 college | 33.0% | | self-taught | 30.6% | | cs major | 34.0% | | non-cs major | 31.8% |
gender gap: 1.8 pp (within noise, down from 3.1). college-tier gap: 0.9 pp (within noise, down from 2.9). cs-vs-non-cs gap: 2.2 pp (within noise, down from 2.7).
the gender gap closed. the college-tier gap closed. the cs-vs-non-cs gap held steady. the overall pass-rate went up.
inter-rater reliability
we scored 60 cohort-2 interviews with both the AI reviewer and a human reviewer, and compared the rubric scores. weighted Cohen's kappa on the binary pass/fail decision: 0.81. on the per-dimension 1–4 scores: 0.72 (specificity), 0.74 (cost), 0.69 (revisit), 0.66 (transfer).
the weakest dimension is "transfer" — predicting what the next decision will be. this is the dimension where the human and AI disagreed most often, and the dimension where the rubric itself is least stable across reviewers. we are considering two changes: (1) raise the bar for "transfer" — require a specific next-decision statement, not a general principle; (2) drop "transfer" entirely from the rubric and make it a 3-dimension rubric with higher cutoffs on the remaining three.
the three changes we made after cohort 1
change 1: tighten "specificity" definition. cohort 1 had applicants scoring 3 on specificity with answers that named a generic principle (e.g., "i used caching"). cohort 2 score-3 answers all named a specific decision (e.g., "i cached the user object in redis with a 60-second TTL keyed on user id"). this raised the noise floor on dimension 1 and reduced the spread of low-information answers.
change 2: raise the cutoff from 10 to 11. cohort 1 passed 38% of applicants at cutoff 10. cohort 2 passed 33% at cutoff 11. the higher cutoff is closer to what the rubric was supposed to measure.
change 3: separate the AI score from the human score. in cohort 1, the AI and the human gave a single joint score. in cohort 2, they score independently, and disagreement >1 on any dimension is escalated to a third reviewer. the third reviewer resolved 11 of 60 cases (18%) in the direction of the AI, 7 (12%) in the direction of the human, and 42 (70%) agreed with both. escalation is now part of the protocol.
what we are still watching
- the cs-vs-non-cs gap. cohort 1 and 2 both showed a small gap, within noise but consistent. we are not yet ready to call it a signal, but we are tracking it.
- the "transfer" dimension. if the human-AI kappa does not improve, we will drop it from the rubric in cohort 3.
- self-taught applicants. both cohorts showed a 2-3 pp pass-rate gap for self-taught applicants. the gap is within noise. we are increasing the self-taught sample in cohort 3 to see if it persists.
we will publish the cohort 3 audit in september 2026. if the gaps do not close further, we will publish a public write-up of what we tried, what worked, and what did not.
the audit is not a defense. it is a check. if the data shows we are biased, we will change the rubric. if the data shows we are not, we will keep the rubric. either way, the data is public.
— dreamclerk team, chennai, may 2026