What Is Wrong With Individual Differences Research?

The Question

Individual differences research, the field that includes personality psychology, psychometrics, and much of applied selection science, rests on some foundational assumptions about measurement that its own measurement specialists have long questioned. Cooper (2024) brings four major but largely unrecognised problems into focus.

The Four Problems

The first is foundational. Measurement specialists have argued that rating scales and ability scales cannot, in principle, measure individual differences in the way the field assumes. Physical measurement involves quantities with known and stable units. Psychological rating scales do not: the intervals between response options are not equal, the same response may mean different things to different people, and the resulting numbers cannot support the arithmetic operations routinely applied to them. Some critics have called this a pathology of science (Cooper, 2024).

The second problem concerns data quality in online research. Online data collection now dominates the field, and it is known to produce seriously flawed results unless careful precautions are taken. Inattentive responding, financial incentive effects on response patterns, and automated bot systems masquerading as human participants have all been shown to substantially affect data quality, yet many published studies have not taken adequate steps to address these threats (Cooper, 2024).

The third problem concerns scale development practices. Some scales have been developed through the repeated paraphrasing of the same item: a single underlying idea is restated in multiple ways, and the resulting items are treated as independent measures. This produces artificially high internal consistency that reflects semantic overlap rather than genuine measurement of a coherent construct (Cooper, 2024).

The fourth problem is arguably the most damaging for the field’s interpretation of its findings. Scales supposedly measuring quite different traits often contain items with almost identical meanings. This artificially inflates the correlations between scales, producing apparent relationships between constructs that may be substantially or entirely artefactual (Cooper, 2024). Models and theories built on these inflated correlations inherit the artefact.

Why It Matters

These are not peripheral methodological concerns. They go to the heart of whether individual differences research is measuring what it claims to measure and whether the relationships it reports reflect genuine empirical regularities or the consequences of how constructs are operationalised. For practitioners who rely on psychometric tools developed within this tradition, the implications deserve serious attention (Cooper, 2024).

Reference

Cooper, C. (2024). What is wrong with individual differences research? Personality and Individual Differences, 221, 112550. https://www.sciencedirect.com/science/article/pii/S0191886924000102