One of the dirty little secrets in the assessment business is the way that assessments are validated. There are no formal systematic approaches for the validation of tests that are easy for tests users, not versed in statistics, to follow. While principles and guidelines exist, these are often not supported by testing companies and are commonly unknown to practitioners.
I have seen every trick in the book pass as validation methods for occupational assessments. Common practices include:
- Making the dependant and independent variables similar (i.e. measuring similar constructs), so that the correlation between the measures increases.
- Having multiple correlations in a single study, and not making Bonferroni adjustment, as a simple means of increasing the odds that the fishing exercise will produce positive results.
- Using low levels of correlation (such as less than 0.1) and stating that the test is now valid. More importantly, effect sizes, on their own, don’t constitute validity, a point I have made earlier.
Most importantly, test publishers rarely follow the simplest of scientific conventions, namely having a hypothesis, and then testing this hypothesis, rather than reengineering their interpretation, post hoc, to better reflect their findings. Researchers then report results in non-quantitative terms, and this is deemed satisfactory by the consumers of these products, which are often HR professionals, with limited statistical or scientific training.
The onus is on us as a profession to rectify this problem, and a recent article in the Journal of Personality Assessment has laid down this specific challenge. While the focus of the article was the Rorschach, the irony is that tests such as the Rorschach and, the other testing often critiqued, the MBTI, have come under the profession’s scrutiny allowing other test publishers to run rampant.
The sooner that we have guidelines that are both clear for test publishers and allow test users to determine the usefulness quickly, or otherwise, of an assessment, the better. Criterion validity is where the rubber meets the road for test usage, and the absence of a sophisticated framework for the validation of psychological tests is an essential gap in the industry.
References
Society for Industrial, Organizational Psychology (US), & American Psychological Association. Division of Industrial-Organizational Psychology. (2018). Principles for the validation and use of personnel selection procedures.
Mihura, J.L., Bombel, G., Dumitrascu, N., Roy, M., & Meadows, E.A. (2019). Why we need a formal systematic approach to validating psychological tests: The case of the Rorschach Comprehensive System. Journal of Personality Assessment, 101, 4, 374-392.
