Why Forecasting is So Unpredictable

Michael Lewis's book, The Undoing Project, sets out why it is hard to predict life and financial outcomes.  The book focuses on the unusual relationship forged between two Israeli psychologists Daniel Kahneman and Amos Tversky. The pair wrote a series of original studies regarding our assumptions about our decision-making process. Their papers showed how the human mind erred systematically when forced to make judgments in uncertain situations. Their work helped to create the field of behavioral economics.

One aspect of the field of behavioral economics focuses on human biases. For example, economics can lay out a brilliant synopsis of what a person should do when making a financial or economic decision, but that doesn't mean the person will choose the best decision.  This is where cognitive biases that can affect human decision-making come into play.  These tendencies to think in certain ways can distort how we view the world around us and affect our judgment. 

Our Most Common Biases

Below are just a few of the most prevalent biases that affect our judgments. A longer list of cognitive biases can be found at The Big Picture and Wikipedia;

• Bandwagon effect – the tendency to do (or believe) things because many other people do (or believe) the same. Related to groupthink, herd behavior, and manias. Carl Jung pioneered the idea of the collective unconscious which is considered by Jungian psychologists to be responsible for this cognitive bias.
• Confirmation bias – the tendency to search for or interpret information in a way that confirms one’s preconceptions.
• Endowment effect – the tendency for people to value something more as soon as they own it.
• Focusing effect – prediction bias occurring when people place too much importance on one aspect of an event; causes an error in accurately predicting the utility of a future outcome.
• Illusion of control – the tendency for human beings to believe they can control or at least influence outcomes which they clearly cannot.
• Loss aversion – the tendency for people to strongly prefer avoiding losses over acquiring gains (see also sunk cost effects)

No matter how great an algorithm or forecasting model you build, you are dealing with irrational humans. This means that you will never be able to predict the outcome of political referendums, elections, and the direction of the stock markets.

Even as great and smart as Kahneman and Tversky have been on this subject, they still had to deal with their own biases just like everyone else.  Having systems and processes in place to help make decisions can overcome some of these natural biases.  Simple things, such as checklists and being mindful of your surroundings, can have life-changing effects.