There appears to be a resurgence of ipsative testing under the veil of CAT (Computer Adaptive Testing) for personality. Clearly, there is a lot more quality research that must go on to develop a CAT for personality than the simple ipsative tests the market has seen before. In regards to CAT testing, the development of item banks where items are calibrated against a latent variable for that attribute, in terms of “how much” attribute it contains is no small task. In addition, CAT testing will clearly speed up the time to complete the test as fewer items are used to make an estimation of personality.
However the problems of ipsative testing remain in that, again, we have a within-person measure used to make comparisons across people. Moreover, the problem is potentially compounded with CAT testing as the actual construct (e.g. trait being measured) is defined by a much smaller subset of items. Both these issues should raise major concerns for test users.
There is no evidence of increased validity (cf. Cherneyshenko, O.S., Stark, S., Drasgow, F., & Roberts, B.W. (2007) Constructing personality scales under the assumption of an Ideal Point response process: toward increasing the flexibility of personality measures. Psychological Assessment, 19, 1, 88-106) and ironically the construct validity is achieved by comparing the results with a standard measure and showing comparability. With these two points in mind, the basis for CAT personality testing is spurious and likely to be underpinned more by a marketing ploy than a commitment to furthering the I/O psychology discipline.
The argument that the value of the new models is seen in terms of administration flexibility and technology rather than in terms of increased predictive validity is worrisome. What this does is send a message to the market that personality can be used with a high degree of accuracy for screening. We know that this is not the case. While screening out at lower levels has merit, the reality remains that personality tests have low criterion validity. The constructs are far more complex from a measurement perspective than ‘g’. Thus, to present a ‘new wave’ of assessments in this regard is very very dangerous as it removes the developmental and coaching component inherent in personality testing.
Arguments around faking are also spurious. As shown in the white paper by Hammond and Barrett (1995): The Psychometric and Practical Implications of the use of Ipsative, forced-choice format, Questionnaires, as within-person measures have an in-built bias for faking. All tests scores contain error and with personality testing error is contributed to by motivated distortion. This is not solved by ipsative testing but rather quite the opposite in that error and distortion becomes harder to detect.
The I/O industry must critique these new innovations on the basis of the science that underpins them. The testing industry is highly competitive and as a result, everyone is looking for a marketing edge. That is what these things are and frankly, in an industry based on science, it demonstrates how market drivers do not necessarily lead to advancement. So long as practitioners still use such questionnaires to measure personality then the issues will remain. These are self-report measures of behaviour and the similarities between tests will always outweigh their differences. Test providers know this and use this as the very basis for the constructs they are measuring. In this regard any attempts at innovation must be seen in the light of the ‘marketing ploy’ it is attempting to deliver.
I will be proud to congratulate real innovation in our field. CAT testing for narrow constructs like ‘g’ makes real sense and has been around for years. In personality, the real interesting work is looking for biological correlates of personality models and other truly revolutionary ways of understanding the personality space of human behaviour. Marketing under the guise of innovation has underpinned I/O psychology since the advent of the commercialisation of research ideas in this space. Let us never confuse innovation with marketing in this discipline anymore and hopefully, the discipline will finally make some real advancement on the understanding of performance.
