Two supervisors can watch the same session and score it differently. One is generous with people they like. One never gives a low rating. One scores the last two minutes. Without a fixed point to measure against, nobody can show which of those is happening — and competency records get signed anyway.
WHAT
The determination is sealed before a human sees it
A completed session is scored against your published rubric using the behavioral evidence. That evidence is what was said and when, the intervention and its effect, and whether protocol was followed. It also counts recovery and critical misses. That determination is recorded and timestamped. Only then does a supervisor review it.
WHY
Impression arrives second, not first
A reviewer cannot be anchored by a number they have not seen. It does not make anybody objective; it changes the order, so the evidence-based determination exists independently of the review rather than downstream of it.
HOW
Then it measures the assessment itself
The supervisor’s score is compared against the same evidence. Over enough independent comparisons that produces a second record: not how reliably staff perform, but how reliably they are assessed.
Evidence,
then opinion
The sequence is the whole claim.
Not “our AI is fairer than your supervisor”. The machine determination and the human one are made independently, against identical evidence, and the gap between them is the thing worth looking at.
3
Rater effects it detects today.
Possible leniency, severity and rubric drift, each reported with the count behind it and never as a verdict about a person. Not yet detected: halo and recency.
8
Independent comparisons before it says anything.
Below that, an evaluator profile would describe the sessions they happened to be given rather than how they assess. A comparison made after they saw the system answer is excluded entirely.
What is different here, and the claim we will not make
What already exists. Checking raters is not a new idea. The INACSL simulation standards recommend testing inter-rater reliability, and at least one major healthcare simulation platform reports how each evaluator’s scores compare with their colleagues’. That comparison has a blind spot: if a whole team scores generously, every evaluator looks consistent. Here, each evaluator is compared with a determination made from the session’s own evidence and sealed before anyone reviewed it, so a habit the whole team shares still shows.
What we will not say. We do not claim to remove bias, and we never will. Automated scoring inherits whatever is in the ratings and rules behind it — this engine included. A platform that exempts itself from that is one study away from being embarrassing. So the claim is narrow on purpose: competency is measured against predefined behavioral evidence before a trainer’s impression enters the process. That is a claim about order, and anyone can check it.
And the wording is enforced. The build fails on any phrasing that says bias has been eliminated, or that calls an evaluator biased. Variance is a measurement of a process; the other is an accusation about a person that a training platform has no standing to make. How it works, in full →
The method is not specific to youth services. Only youth behavioral health is built.
Assessment Integrity asks the same five questions in any setting: which competency was assessed, what happened, what the organization counts as acceptable, what the evaluator scored, and how far apart those are. So the assessment layer holds no competency framework of its own; the framework and the terminology are settings. These are the fields it is configured for today.
BuiltYouth behavioral health
Assesses a member of staff working with a young person.
Not built yetHealthcare and nursing
Assesses a clinician working with a patient.
Not built yetChild welfare
Assesses a caseworker working with a parent or caregiver.
Not built yetCrisis response
Assesses a responder working with a person in crisis.
Not built yetJuvenile justice
Assesses an officer working with a young person in custody.
Youth behavioral health has the scenarios, the personas, the behavioral model and a competency framework. The other 4 fields have none of those yet, and each lists what would have to be written before it could be offered. What each field would need →