One of the myths I have long spoken about is the idea that there are great differences between various psychological tests. In essence, the argument is that if there is a science to personality, or cognitive ability, what they measure must be similar. This is fundamental to the basis for construct validity between tests, i.e. what they claim to measure. I was recently introduced to a paper that explored this exact issue. Not surprisingly, the similarities between similar tools are far greater than their differences.

Grucza, R.A., Goldberg, L.R. (2007) The Comparative Validity of 11 Modern Personality Inventories: Predictions of Behavioral Acts, Informant Reports, and Clinical Indicators. Journal of Personality Assessment, 89, 2, 167-187.

Abstract
In science, multiple measures of the same constructs can be useful, but they are unlikely to all be equally valid indicators. In psychological assessment, the many popular personality inventories available in the marketplace also may be useful, but their comparative validity has long remained unassessed. This is the first comprehensive comparison of 11 such multi-scale instruments against each of three types of criteria: clusters of behavioral acts, descriptions by knowledgeable informants, and clinical indicators potentially associated with various types of psychopathology. Using 1,000 bootstrap resampling analyses from a sample of roughly 700 adult research participants, we assess the relative predictability of each criterion and the comparative validity of each inventory. Although there was a wide range of criterion predictability, most inventories exhibited quite similar cross-validities when averaged across all three types of criteria. On the other hand, there were important differences between inventories in their predictive capabilities for particular criteria. We discuss the factors that lead to differential validity across predictors and criteria. The tests used were:

1. The revised NEO inventory (NEO-PI-R: Costa & McCrae, 1992)
2. The California Psychological Inventory (CPI: Gough & Bradley, 2002)
3. The 5th edition of Cattell’s Sixteen Personality Factor Questionnaire (16PF: Conn & Rieke, 1994; Russell & Karol, 1994)
4. The Hogan Personality Inventory (HPI: Hogan & Hogan, 1995)
5. The Temperament and Character Inventory (TCI-R: Cloninger, Przybeck, Svrakic, & Wetzel, 1994)
6. The Multidimensional Personality Questionnaire (MPQ: Tellegen, in press; Tellegen & Waller, in press)
7. The revised version of the Jackson Personality Inventory (JPI-R: Jackson, 1994)
8. The Six-Factor Personality Questionnaire (6FPQ: Jackson, Paunonen, & Tremblay, 2000)
9. The HEXACO Personality Inventory (HEXACO-PI: Lee & Ashton, 2004)
10.The set of 100 unipolar adjective markers of the Big-Five factor structure developed by Goldberg (1992)
11.The 485 items in the IPIP-AB5C Inventory (Goldberg,1999a)

I believe that if we are to progress, what is required is a recognition of a fundamental science that underpins the discipline. Measures must be related back to that science. No other discipline has this great variation in taxonomies of measurement nor should I/O psychology.