Psychometric foundations: consistency, evidence, accuracy and fairness

Notions clés

Reliability and the standard error of measurement

Reliability is the consistency of scores across occasions, forms, items or raters. The standard error of measurement (SEM), the standard deviation times the square root of (1 minus reliability), sets the band of error around one person's score; higher reliability gives a narrower band.

Validity as evidence for a use

Validity is the degree to which evidence supports a specific interpretation of scores for a proposed use, not a fixed property of the test. The Standards name five sources of evidence: test content, response processes, internal structure, relations to other variables, and the consequences of testing.

Item analysis and standardization

Item analysis checks each item's difficulty (the proportion of examinees answering correctly) and discrimination (how well it separates high from low scorers). Standardization fixes administration and scoring so that norms gathered from a representative sample can be applied to a new examinee.

Sensitivity, specificity and predictive value

Sensitivity is the share of people with a condition who test positive; specificity is the share without it who test negative. Predictive values also depend on the base rate (how common the condition is in the group tested), so when a condition is rare, many positive results are false.

Fairness and bias

A difference in group means is not, by itself, bias. Bias is construct-irrelevant: items that function differently for groups matched on the trait (differential item functioning), or the same score predicting different outcomes in different groups (differential prediction).

Exemple commenté

A school district gives a brief reading screener to 500 first graders. Full evaluations later show that 50 of them have a reading disorder. The screener's sensitivity is .80 and its specificity is .90, and the district wants to know what a positive screen means for a parent.

1. Children with the disorder: 50. Sensitivity of .80 flags 40 of them (true positives) and misses 10 (false negatives).
2. Children without it: 450. Specificity of .90 clears 405 (true negatives) and flags 45 (false positives).
3. Positive predictive value is 40 of the 85 flagged children, about 47 percent: fewer than half of the flagged children have the disorder.
4. Negative predictive value is 405 of the 415 cleared children, about 98 percent, so a negative screen is reassuring, while a positive screen calls for a full evaluation rather than a label.

Sensitivity and specificity describe the test; predictive values describe what a result means in a particular group, and they shift with the base rate.