How Common Is It?

Nadia loves birds. In her park, most small brown birds are sparrows, but a rare bird called a lark sometimes visits. Nadia has a phone app that listens to a bird and names it. It is a good app: it is right 9 times out of 10. One morning a small brown bird sings, and the app says "Lark!" Nadia is thrilled. How likely is it that she has really found a lark?

The answer most people give

Most people say: very likely - the app is right 9 times in 10, so it's about 9 in 10 that this is a lark. It sounds right. It is badly wrong. To see why, you need one more fact that Nadia forgot to ask about: how common are larks in this park in the first place?

Counting 100 birds

Say that out of every 100 small brown birds in the park, 97 are sparrows and only 3 are larks. Now imagine the app listening to all 100.

Of the 3 larks, the app gets almost all of them right: it says "Lark!" for about 3.
Of the 97 sparrows, the app is wrong 1 time in 10. That means it wrongly says "Lark!" for about 10 of them.

So the app says "Lark!" about 13 times, and only about 3 of those are really larks. When the app says "Lark!", it is right only about 3 times in 13 - less than 1 in 4. Nadia's bird is still most likely a sparrow.

A grid of 100 small bird drawings, 10 rows of 10. 97 are brown sparrows and 3 are golden larks. Small red "Lark!" tags hang on the 3 larks and on 10 of the sparrows. Underneath, the tagged birds are gathered into one group of 13, with only 3 larks in it

How common it is

The number Nadia forgot is called the base rate: how common something is in the first place, before you look at the sign. The psychologists Daniel Kahneman and Amos Tversky showed in the 1970s that people very often forget it, even clever people and experts.

The lesson is not that the app is bad. It is right 9 times in 10. The lesson is that when the thing you are looking for is rare, most of the "yes" answers from a good-but-not-perfect test will come from the many ordinary cases, not the few rare ones. There are simply so many more sparrows to get wrong.

Two jars of marbles side by side. One has almost all brown marbles and only 3 gold ones; the other has half brown and half gold. Above both, a question: "Picked a gold one?"

What to do with it

When a sign says "yes, it's the rare thing", ask two questions, not one:

How often is this sign right?
How common is the thing in the first place?

If the thing is rare, check again in another way before you get excited. Nadia could take a photo, or wait for the bird to sing again - two independent signs pointing the same way are much stronger than one.

Try it at home

A smoke alarm in a kitchen goes off. Is a fire likely? How common are fires compared with burnt toast?
Your friend's phone says "Possible spam" about a message from their grandmother. Should they believe it?
Think of a rare thing you once got excited about too quickly.

Words to know

base rate - how common something is in the first place, before you look at any sign
rare - happening only in a few cases out of many

Big idea

Before you trust a sign, ask how common the thing is. When it is rare, even a good test gives more false alarms than true ones - so check again before you believe it.