Keeping score

About this video

A weather forecaster says there's a 7 in 10 chance of rain tomorrow, and it stays dry. Was she wrong? You can't tell from one day. But collect every day of a year when she said 7 in 10: if it rained on about 7 of every 10 of those days, she was doing her job perfectly. If it rained on only 3 of every 10, something was off. You can check yourself the same way.

Belief comes in degrees, from no idea to sure, and a careful thinker is as sure as the evidence allows. Here is a way to test whether you manage it. Take all the times you said you were 9 out of 10 sure: if you are careful, about 9 in 10 of them should turn out true. Take all the times you were 6 out of 10 sure: about 6 in 10 should be true. When your hits match your confidence like this, you are calibrated, a word borrowed from measuring tools set to read correctly. Weather forecasters are among the best-calibrated people we know of, because they forecast every day and find out the next day whether they were right.

Most of us are not like forecasters. When psychologists ask people questions and also ask how sure they are, people are usually too sure: of the answers people call certain, a surprising number are wrong. This is overconfidence. We rarely keep score, we remember the times we were right, and feeling sure comes from how easily an answer comes to mind, not from how good our reasons are. You can train yourself the way forecasters do: when you guess, say how sure you are, write it down, and later check and count. If your 9 out of 10 answers are right only 6 times in 10, you are too sure, and now you know by how much. This video goes with the How Often Are You Right? lesson of Knowledge Level 2 on Knaph.

Transcript

1. 0:00Seven in ten chance of rain, says the forecaster. It stays dry. Was she wrong?
2. 0:05One day can't tell. Collect every seven-in-ten day of a year.
3. 0:11If it rained on about seven in ten of them, she's calibrated.
4. 0:17Most of us are too sure. Our certain answers are often wrong.
5. 0:23Keep score. Say how sure you are, write it down, then check.
6. 0:29Of your nine-in-ten answers, are nine in ten true?