How Often Are You Right?

A weather forecaster says: "Tomorrow there's a 7 in 10 chance of rain." It stays dry. Was she wrong? You can't tell from one day. But suppose you collect every day for a year when she said "7 in 10". If it rained on about 7 of every 10 of those days, she was doing her job perfectly. If it rained on only 3 of every 10, something was off. How could you check yourself the same way?

Right as often as you are sure

In Knowledge Level 1 we saw that belief comes in degrees, from "no idea" to "sure", and that a careful thinker is as sure as the evidence allows. Here is a way to test whether you manage it.

Take all the times you said you were "9 out of 10 sure". If you are careful, about 9 in 10 of them should turn out true. Take all the times you were "6 out of 10 sure": about 6 in 10 should be true. When your hits match your confidence like this, you are calibrated, a word borrowed from measuring tools that are set to read correctly.

Weather forecasters are among the best-calibrated people we know of. Researchers checked thousands of their forecasts and found that when they said "7 in 10", it rained on about 7 days in 10. They get this good because they make a forecast every day and find out the next day if they were right.

Too sure

Most of us are not like forecasters. When psychologists ask people questions and also ask how sure they are, people are usually too sure. Of the answers people call "certain", a surprising number are wrong. This is called overconfidence.

Why? We rarely keep score. We remember the times we were right and forget the others (as in the I Knew It All Along lesson). And feeling sure is easy: it comes from how easily an answer comes to mind (as in the What Comes to Mind lesson), not from how good our reasons are.

Keeping score

You can train yourself the way forecasters do:

1. When you make a guess, also say how sure you are: "7 out of 10".
2. Write it down.
3. Later, check. After many guesses, count: of your "7 out of 10" answers, how many were right?

If your "9 out of 10" answers are right only 6 times in 10, you are too sure - and now you know by how much.

Try it at home

1. Write down ten facts you could check (a country's capital, a friend's birthday, a year). For each one, write how sure you are, from 5 to 10 out of 10.
2. Check them all.
3. Did the ones you were surest of come out best? Were you too sure, about right, or not sure enough?

Words to know

• calibrated - right about as often as you are sure: of your "8 out of 10" answers, about 8 in 10 are true
• overconfidence - being surer than your record of being right allows
• keeping score - writing down guesses and how sure you were, then checking

Big idea

Confidence can be tested. Of all the things you are "9 out of 10 sure" about, about 9 in 10 should be true. Most people are too sure - keeping score, as weather forecasters do, is how you find out and get better.