Validate your AI hiring tools before a court does it for you.
If an AI scores, ranks or screens candidates for you, you need more than a bias audit. You need evidence that the scores are reliable, valid and defensible. I/O psychologists have done this for selection tests for fifty years. I do it for AI raters.
A bias audit tells you who got screened out. It doesn't tell you whether the tool works.
Most employers using AI in hiring have one piece of paper: an adverse-impact summary, often supplied by the vendor. It reports selection rates by race and sex. It says nothing about whether the scores predict anything.
That gap is where the risk now sits. Under the Uniform Guidelines, a tool that shows adverse impact must be defended as job-related, and that defense is validity evidence. In Mobley v. Workday, discovery is where scoring data and audit history come into the open. New York City's enforcement of its AI hiring law tightened in 2026, and the EU's high-risk employment rules take effect in December 2027.
LLM-based scorers make it harder. Their output changes with the model version, the prompt and the sampling settings, and their stated confidence runs high. Few have been tested the way a structured interview is tested before it's allowed to touch a hiring decision.
The question counsel will eventually ask is simple: show me the validation study. This page is about having one.
An AI scorer is a new rater. It gets the questions we ask any rater.
Mapped to SIOP's 2023 Considerations and Recommendations for the Validation and Use of AI-Based Assessments, so your file cites the standard that experts on both sides will recognize.
| Question | SIOP 2023 | What you get |
|---|---|---|
| Does it agree with itself? | §2 consistency across repeats and raters | Test-retest across runs; agreement with trained human raters; agreement across model versions |
| Does it predict anything? | §1.1.1, §4.3.5 criterion-related validity | Prediction of performance, training or retention outcomes, with intervals, on your real applicant flow |
| Does it add anything? | §4.6.2 increments in validity | Incremental validity over your existing interview or screen. If it adds nothing, you find out before paying for another year |
| Is it fair? | §3.2 predictive and measurement bias | Cleary-model slope and intercept tests by protected group, separate from adverse-impact ratios |
| Does it stay valid? | §4.3.2, §4.4.2, §4.7.2 algorithm change and drift | Version-locked scoring, drift monitoring by period and job family, and a rule for when a vendor update triggers re-validation |
| Can anyone check it? | §5 documentation for external audit | A technical report another I/O psychologist could reproduce, written to the Uniform Guidelines and the SIOP Principles |
The deliverable is the report plus a short summary for counsel. Vendor-supplied audits are reviewed, not assumed.
Five things the guidelines don't ask yet. I test them anyway.
SIOP's AI guidance was finished in January 2023, weeks after large language models reached the public. It treats AI as a trained model with fixed features. LLM scorers don't behave that way. Each item below shows how the live ThePickLog study handles it.
1. Calibration
When the tool says 70%, does that happen 70% of the time? LLM confidence tends to run high, so raw and recalibrated calibration are both reported.
ThePickLog: Platt recalibration on the first 100 picks, holdout ECE ≤ 0.102. The prompt is part of the rater
A change to model, prompt or settings creates a new rater, not an update. Every score is tied to the exact configuration that produced it.
ThePickLog: judge id = hash of vendor, model, prompt, arm3. Randomness is measurement error
Ask an LLM the same question three times and you may get three answers. That is reliability, and it is measured, not assumed.
ThePickLog: k = 3 repeats, ICC(1) ≥ 0.804. Knowing when to defer
A rater should be able to say "I don't know." The test is whether its abstentions land on its worst cases, without letting abstention hide failures.
ThePickLog: an abstention is scored as the incumbent's forecast5. Explanations are not evidence
AI-generated reasons are logged and coded but never used to decide whether the tool passes. Model explanations are often not the real basis for the score.
ThePickLog: reasons reported, not scoredI do this in public, and I publish the losses.
I run ThePickLog, a public research lab that tests prediction claims the way a selection instrument is tested: the rule is frozen with a date before the data exist, graded forward against a baseline, and the verdict is published whether it passes or fails. The first claim tested was my own. It failed, significantly, and the full write-up is on the site.
On October 25, 2026, the lab pre-registered a validation of two AI raters on exactly the terms on this page. Each reads the same pre-open SEC filings and forecasts whether a stock will fall 20% within six sessions. Both are scored on reliability, calibration, discrimination and incremental validity over the existing score. The thresholds were frozen before any prediction was made. One verdict, on or after January 11, 2027, published either way.
That is the standard I bring to your tool.
H-JUDGE1: can an AI judge forecast drawdown better than the score already does?
- Registered
- 2026-10-25, before the window opened
- Raters
- Jev 1.13.0 and OpenAI gpt-5.5-2026-04-23, both version-pinned
- Design
- 3 input arms, including a null control and a masked-identity arm; 3 repeats each
- Pass bar
- Incremental AUC ≥ 0.03 over the score, Holm-adjusted across 6 tests; ICC ≥ 0.80; calibrated after recalibration
- Verdict
- One look, on or after 2027-01-11
| Trades graded | Edge vs baseline | What it looked like |
|---|---|---|
| 30 | +1.7 pp | "It's working." |
| 200 | −2.1 pp | "That's concerning." |
| 309 | −3.0 pp | Loses, significantly |
95% CI [−4.4, −1.5], company-clustered. Read the verdict.
Three buyers, one report.
Using AI to screen resumes, score interviews, rank candidates or inform promotion, especially those hiring into New York City or the EU, or with an applicant history on a platform now in litigation.
Who need an independent technical report their customers' counsel will accept. Fixed fee, paid before the readout, and the result is reported as found.
Preparing a privileged review and needing the technical work done by a credentialed I/O psychologist.
Thresholds set before the data. Then the work.
- Scoping call. What the tool does, which decisions it touches, what evidence exists. A written gap read within a week, at no charge.
- Study design. Criterion, sample, subgroups and pass thresholds, documented and dated before any outcome data are analyzed, so nobody can move them afterward, including me.
- Data and analysis. Usually 6 to 12 weeks. Your data stays in your environment where possible.
- Technical report and counsel summary. Reproducible by another I/O psychologist.
- Monitoring. What triggers re-validation, and a standing drift check so the next vendor update doesn't quietly invalidate the file.
Studies are priced by scope. A monitoring retainer is available after the initial study.
Josh Davis, Ph.D.
I hold a Ph.D. in Industrial/Organizational Psychology from the University of Oklahoma, with a minor in quantitative psychology, where I trained under Michael Mumford. For more than fifteen years I've built and validated executive assessment and selection systems for enterprise clients in defense, automotive, technology, financial services, pharmaceuticals and energy. I'm Hogan certified, and my published research includes work on how people regulate emotion under workplace stress.
I'm not an AI vendor, and I don't sell hiring tools. My interest in AI raters is the one I've always had: does this measure predict what it claims to, or does it only look like it does?
The question isn't whether your AI hiring tool will be examined. It's whether you'll have the study when it is.
Request a scoping callOr email sales@amdconsultingservices.com with the name of the tool and the decisions it touches.