Superforecasting: The Art and Science of Prediction

It seems only logical that I kick off this new series of recommendations with a book called Superforcasting. It was this very book that started my curiosity into data (science). And it really helped me with thinking more precisely, so let’s dive into the reason why I think you should read this book.

But, before I continue about Superforecasting and perhaps how to become one yourself, I want to emphasize that this book is so much more. The tips about becoming a forecaster are in my opinion transferable to becoming a good scientist and perhaps becoming a good person in general.

Things like asking the right questions, being aware of your flaws, having a reasonable amount of doubt, and looking for different perspectives, are all qualities every person should have, which would make the world a better place.

A small history of fortunity

Fortune cookies and Nostradamus

Let’s start with dipping our toes into fortunity, forecasting, predictions and prophecies. This area was always surrounded by magic and mystery. Nobody knew before what was coming. We (as humans) were praying to gods for good weather. We went to fortune tellers and psychics to learn about our path in life. We even order Chinese food, just to get a fortune cookie.

Sports betting wasn’t a real thing. Back in the days we hadn’t a clue. The only people who were good at predicting outcomes in sports matches were the people involved in match fixing. The first game that was fixed was the now famous Canton Bulldogs–Massillon Tigers betting scandal in 1906.

The Greek poet named Archilochus wrote the next sentence: “Πόλλ᾽ οἶδ᾽ ἀλώπηξ, ἀλλ’ ἐχῖνος ἕν μέγα,” which roughly translates to “The fox knows many things, but the hedgehog knows one big thing.”

I have been thinking about this quote a lot these past few weeks while reading this book, especially since it is often cited in forecasting literature, and therefor also in Superforecasting. It sometimes seems as if we are increasingly surrounded by hedgehogs. The majority follows one big idea and can therefore be delightfully un-nuanced.

And I understand this behaviour it in a way, because having one (big) idea and following it firmly is easier than always looking for nuance. Searching for shades of gray and trying to puncture your own opinion, is hard.

And It is also true that hedgehogs make tv/podcasts/blogs that are more fun to watch/hear/read, but the predictions are bad…

It started with an experiment

Good Judgement Project

Tetlock studied the hedgehogs on TV and found something astonishing: the average expert could predict no better than a dart-throwing chimpanzee. He had studied three hundred people who made predictions their job, and… they might as well have been tossing a coin.

Tetlock started an experiment and found out that there was a small group that predicted better than the chimpanzee. They were not oracles, that was for sure, but they did a slightly less crappy job than the rest. So he decided to gather a team of volunteers to participate in the largest prediction competition ever, organized by the American Intelligent Services.

More specifically, in 2011, IARPA – the US intelligence community – launched a massive competition to identify cutting-edge methods to forecast geopolitical events. Everybody could join this competition, and for a whole year, you had to answer lots of questions.

In the first year of the competition, it was already clear: Tetlock’s team did better than the other teams. Much better. This remained the case for the rest of the competition. Four years, five hundred questions and more than a million predictions later, it was certain. Tetlock had found them: superforecasters. They emerged as the undisputed victor in the tournament. GJP’s predictions were so accurate that they even outperformed intelligence analysts with access to classified data.

These superforecasters weren’t necessarily super smart, nor did they have much understanding of the topics covered in the contest. Their jobs often had little to do with forecasting. There was a filmmaker, a ballroom dancer, a retired computer programmer.

“It’s not really who they are, it’s what they do,” Tetlock says in Superforecasting. Nearly every superforecaster, he found, took a number of steps that anyone can learn. It wasn’t a coincidence or simple luck that they were so good at predicting. They all shared common qualities and behaviours that made them Superforcasters.

A few common qualities

What makes them so good at forecasting?

I will share a few common qualities that all Superforcasters had, but of course, I can’t recommend you enough to just read the book. It’s well-written and spot-on.

Break the problem into manageable, smaller pieces

A question like ‘how many people will die worldwide from a new form of COV-id?’ is too big. Make an overview of all the bits of information you need, such as: ‘Which forms of COV-id are now known?’.

Look at the problem from the outside

Don’t immediately dive into the specifics of a situation, but look at the problem more broadly. Suppose you believe that your side project will be a success. How realistic is that prediction? Take a step back. Ignore the details about your own company and look more broadly: how often does a side project succeed?

Find a good balance between the outside and the inside view

Supplement the outside view with information about the specific situation. What is the chance that Kamala Harris will become the next president of the US? Then you are not only looking at historical data, but also at the current situation. Superforecasters have a good sense of how to weigh information.

Don’t adjust too much, but also not too little

“Updating your beliefs is to good prediction what brushing and flossing are to good dental hygiene,” Tetlock says. “It can be boring, occasionally uncomfortable, but it pays off in the long run.” Don’t jump on every news story, like a tweet, but don’t be too rigid in your ideas either.

Find as many perspectives as possible

This is where the fox comes in. Gather as much information as you can, don’t hesitate to change your prediction and always look for counterarguments to your reasoning. And always keep doubting.

Final thoughts

Things I am thinking about

Although I can’t recommend this book enough for the reasons I said at the beginning of this article, I do wonder whether this forecasting skill has become obsolete. If you’d asked me this question a year ago, I would have said no, because I wasn’t sure computers (or better, algorithms/learning models) could put some reasonable doubt in their decision-making. After all, in the end, computers are binary, 0-1, on-off. There are no shades of grey.

But with the new technology on the horizon and the self-learning models, I do think, computers can become Superforecasters. So I am curious about what you think after you’ve read this book. Do we still need humans for predicting the future or can we let computers do it for us?
Let me know in the comments.