In a recent book, Nate Silver illustrates the limitations of people to think in probabilistic terms and build models that incorporate uncertainty. This is not a trivial matter and is a major contributing factor to the financial crash that started in late 2007, the effects of which are still resonating around the world.
Mr Silver notes that while financial advisers have widely proclaimed the GFC as an unpredictable ‘Black Swan’ event that no one saw coming, he notes that there were plenty of observers, including the newspapers of the day, that were warning of problems brewing. However, those that were close to the action missed the signs as they were primed to avoid seeing disaster and arrogantly had full confidence in their powers of prediction, due to their position.
For I/O psychologists. the level of over-confidence people have in their predictions is a well-known fact. We have long known, for example, that the predictive power of cognitive ability tests exceeds that of interviews while the latter is still more commonly used as the stalwart of the selection process. It is ironic that those tasked with the specific purpose of prediction (selection professionals) are often not skilled in prediction and choose imprecise methodologies. This phenomena of ‘I must be right as I’m the professional’ is common with so many professionals from doctors to lawyers, accountants to bankers. However, when their errors are pointed out to them they take umbrage and get defensive.
Another issue identified by Mr Silver is the idea that data alone can solve problems. This is the “psychometrician fallacy” that somehow the numbers alone will both identify and solve problems. On the contrary, numbers alone without good logic are akin to having the materials of a house and no plan to build it.
Prediction, for I/O psychologists, is a craft. We use tools such as the data provided by psychometric tools, to carefully construct causal models that we then tried to find evidence to refute or support our theory. We must avoid any form of over-confidence, however attractive the commercial benefits are, and always recognise the inherent difficulties in the prediction of human outcomes. We provide benefit to our clients not by being over-confident and assured, but by being honest and providing sound, well-supported advice.
The Signal and the Noise: Why So Many Predictions Fail—But Some Don’t. By Nate Silver. Penguin Press
