How to Lie With Statistics
don’t. These, the records of the Connecticut Tumor Registry, go back
To be worth much, a report based on sampling must use a representative sample, which is one from which every source of bias has been removed.
The basic sample is the kind called “random.” It is selected by pure chance from the “universe,” a word by which the statistician means the whole of which the sample is a part. Every tenth name is pulled from a file of index cards. Fifty slips of paper are taken from a hatful. Every twentieth person met on Market Street is interviewed. (But remember that this last is not a sample of the population of the world, or of the United States, or of San Francisco, but only of the people on Market Street at the time.
No conclusion that “sixty-seven per cent of the American people are against” something or other should be read without the lingering question, Sixty-seven per cent of which American people?
The third question probed attitudes that might be based on feelings revealed by the first two. “Do you think it is more important to concentrate on beating the Axis, or to make democracy work better here at home?” “Beat Axis” was the reply of thirty-nine per cent, according to the black interviewers; of sixty-two per cent, according to the white. Here is bias introduced by unknown factors. It seems likely that the most effective factor was a tendency that must always be allowed for in reading poll results, a desire to give a pleasing answer. Would it be any wonder if, when answering a question with connotations of disloyalty in wartime, a Southern black would tell a white man what sounded good rather than what he actually believed? It is also possible that the different groups of interviewers chose different kinds of people to talk to.
When you are told that something is an average you still don’t know very much about it unless you can find out which of the common kinds of average it is—mean, median, or mode.
The different averages come out close together when you deal with data, such as those having to do with many human characteristics, that have the grace to fall close to what is called the normal distribution.
But three of the inhabitants are millionaire week-enders and these three boost the total income, and therefore the arithmetic average, enormously. They boost it to a figure that practically everybody in the neighborhood has a good deal less than. You have in reality the case that sounds like a joke or a figure of speech: Nearly everybody is below average.
Only when there is a substantial number of trials involved is the law of averages a useful description or prediction.
“Gesell’s norms”
Let his child fail to sit by the specified age and the parent must conclude that his offspring is “retarded” or “subnormal” or something equally invidious. Since half the children are bound to fail to sit by the time mentioned, a good many parents are made unhappy. Of course, speaking mathematically, this unhappiness is balanced by the joy of the other fifty per cent of parents in discovering that their children are “advanced.” But harm can come of the efforts of the unhappy parents to force their children to conform to the norms and thus be backward no longer.
A good deal of the stupid criticism of Dr. Alfred Kinsey’s well-known (if hardly well-read) report came from taking normal to be equivalent to good, right, desirable. Dr. Kinsey was accused of corrupting youth by giving them ideas and particularly by calling all sorts of popular but unapproved sexual practices normal. But he simply said that he had found these activities to be usual, which is what normal means, and he did not stamp them with any seal of approval.
Place little faith in an average or a graph or a trend when those important figures are missing. Otherwise you are as blind as a man choosing a camp site from a report of mean temperature alone. You might take 61 degrees as a comfortable annual mean, giving you a choice in California between such areas as the inland desert and San Nicolas Island off the south coast. But you can freeze or roast if you ignore the range. For San Nicolas it is 47 to 87 degrees but for the desert it is 15 to 104. Oklahoma City can claim a similar average temperature for the last sixty years: 60.2 degrees. But as you can see from the chart below, that cool and comfortable figure conceals a range of 130 degrees.
(Most statisticians now prefer to use another, but comparable, measurement called the standard error. It takes in about two-thirds of the cases instead of exactly half and is considerably handier in a mathematical way. For our purposes we can stick to the probable error, which is the one still used in connection with the Stanford-Binet.)
Sometimes the big ado is made about a difference that is mathematically real and demonstrable but so tiny as to have no importance. This is in defiance of the fine old saying that a difference is a difference only if it makes a difference.
Chop off the bottom. Now that’s more like it. (You’ve saved paper too, something to point out if any carping fellow objects to your misleading graphics.)
I simply draw a moneybag to represent the Rotundian’s thirty dollars, and then I draw another one twice as tall to represent the American’s sixty. That’s in proportion, isn’t it? Now that gives the impression I’m after. The American’s wage now dwarfs the foreigner’s. The catch, of course, is this. Because the second bag is twice as high as the first, it is also twice as wide. It occupies not twice but four times as much area on the page. The numbers still say two to one, but the visual impression, which is the dominating one most of the time, says the ratio is four to one. Or worse. Since these are pictures of objects having in reality three dimensions, the second must also be twice as thick as the first. As your geometry book put it, the volumes of similar solids vary as the cube of any like dimension. Two times two times two is eight. If one moneybag holds $30, the other, having eight times the volume, must hold not $60 but $240.
Some of this may be no more than sloppy draftsmanship. But it is rather like being short-changed: When all the mistakes are in the cashier’s favor, you can’t help wondering.
IF YOU can’t prove what you want to prove, demonstrate something else and pretend that they are the same thing. In the daze that follows the collision of statistics with the human mind, hardly anybody will notice the difference. The semiattached figure is a device guaranteed to stand you in good stead. It always has. You can’t prove that your nostrum cures colds, but you can publish (in large type) a sworn laboratory report that half an ounce of the stuff killed 31,108 germs in a test tube in eleven seconds.
Advertisers aren’t the only people who will fool you with numbers if you let them. An article on driving safety, published by This Week magazine undoubtedly with your best interests at heart, told you what might happen to you if you went “hurtling down the highway at 70 miles an hour, careening from side to side.” You would have, the article said, four times as good a chance of staying alive if the time were seven in the morning than if it were seven at night. The evidence: “Four times more fatalities occur on the highways at 7 P.M. than at 7 A.M.” Now that is approximately true, but the conclusion doesn’t follow. More people are killed in the evening than in the morning simply because more people are on the highways then to be killed.
By the same kind of nonsense that the article writer used you can show that clear weather is more dangerous than foggy weather. More accidents occur in clear weather, because there is more clear weather than foggy weather. All the same, fog may be much more dangerous to drive in.
Equally gay fun is to be had with percentages. For a recent nine-month period General Motors was able to report a relatively modest profit (after taxes) of 12.6 per cent on sales. But for that same period GM’s profit on its investment came to 44.8 per cent, which sounds a good deal worse—or better, depending on what kind of argument you are trying to win. Similarly, a reader of Harper’s magazine came to the defense of the A & P stores in that magazine’s letters column by pointing to low net earnings of only 1.1 per cent on sales. He asked, “Would any American citizen fear public condemnation as a profiteer... for realizing a little over $10 for every $1,000 invested during a year?” Offhand this 1.1 per cent sounds almost distressingly small. Compare it with the four to six per cent or more interest that most of us are familiar with from FHA mortgages and bank loans and such. Wouldn’t the A & P be better off if it went out of the grocery business and put its capital into the bank and lived off interest?
The catch is that annual return on investment is not the same kettle of fish as earnings on total sales. As another reader replied in a later issue of Harper’s, “If I purchase an article every morning for 99 cents and sell it each afternoon for one dollar, I will make only 1 per cent on total sales, but 365 per cent on invested money during the year.”
The death rate in the Navy during the Spanish-American War was nine per thousand. For civilians in New York City during the same period it was sixteen per thousand. Navy recruiters later used these figures to show that it was safer to be in the Navy than out of it. Assume these figures to be accurate, as they probably are. Stop for a moment and see if you can spot what makes them, or at least the conclusion the recruiting people drew from them, virtually meaningless. The groups are not comparable. The Navy is made up mainly of young men in known good health. A civilian population includes infants, the old, and the ill, all of whom have a higher death rate wherever they are. These figures do not at all prove that men meeting Navy standards will live longer in the Navy than out. They do not prove the contrary either.
When Dewey was elected Governor in 1942, the minimum teacher’s salary in some districts was as low as $900 a year. Today the school teachers in New York State enjoy the highest salaries in the world. Upon Governor Dewey’s recommendation, based on the findings of a Committee he appointed, the Legislature in 1947 appropriated $32,000,000 out of a state surplus to provide an immediate increase in the salaries of school teachers. As a result the minimum salaries of teachers in New York City range from $2,500 to $5,325. It is entirely possible that Mr. Dewey has proved himself the teacher’s friend, but these figures don’t show it. It is the old before-and-after trick, with a number of un-mentioned factors introduced and made to appear what they are not. Here you have a “before” of $900 and an “after” of $2,500 to $5,325, which sounds like an improvement indeed. But the small figure is the lowest salary in any rural district of the state, and the big one is the range in New York City alone. There may have been an improvement under Governor Dewey, and there may not.
THE DARKENING SHADOW
For a spurious air of precision that will lend all kinds of weight to the most disreputable statistic, consider the decimal. Ask a hundred citizens how many hours they slept last night. Come out with a total of, say, 783.1. Any such data are far from precise to begin with. Most people will miss their guess by fifteen minutes or more, and there is no assurance that the errors will balance out. We all know someone who will recall five sleepless minutes as half a night of tossing insomnia. But go ahead, do your arithmetic, and announce that people sleep an average of 7.831 hours a night. You will sound as if you knew precisely what you were talking about. If you had been so foolish as to declare only that people sleep 7.8 (or “almost 8”) hours a night, there would have been nothing striking about it. It would have sounded like what it was, a poor approximation and no more instructive than almost anybody’s guess.
Percentages offer a fertile field for confusion. And like the ever-impressive decimal they can lend an aura of precision to the inexact. The United States Department of Labor’s Monthly Labor Review once stated that of the offers of part-time household employment with provisions for carfare, in Washington, D. C., during a specified month, 4.9 per cent were at $18 a week. This percentage, it turned out, was based on precisely two cases, there having been only forty-one offers altogether. Any percentage figure based on a small number of cases is likely to be misleading. It is more informative to give the figure itself. And when the percentage is carried out to decimal places you begin to run the scale from the silly to the fraudulent.
You can check on this bit of statistical misfiguring by supposing, for simplicity, that the original wage was $1 an hour. Cut twenty per cent, it is down to 80 cents. A five per cent increase on that is 4 cents, which is not one-fourth but one-fifth of the cut. Like so many presumably honest mistakes, this one somehow managed to come out an exaggeration which made a better story. All this illustrates why to offset a pay cut of fifty per cent you must get a raise of one hundred per cent.
It was the Times also that once reported that, for a fiscal year, air mail “lost through fire was 4,863 pounds, or a percentage of but 0.00063.” The story said that planes had carried 7,715,741 pounds of mail during the year. An insurance company basing its rates in that way could get into a pack of trouble. Figure the loss and you’ll find that it came to 0.063 per cent or one hundred times as great as the newspaper had it.
It is the illusion of the shifting base that accounts for the trickiness of adding discounts. When a hardware jobber offers “50% and 20% off list,” he doesn’t mean a seventy per cent discount. The cut is sixty per cent since the twenty per cent is figured on the smaller base left after taking off fifty per cent. A good deal of bumbling and chicanery have come from adding together things that don’t add up but merely seem to. Children for generations have been using a form of this device to prove that they don’t go to school. You probably recall it. Starting with 365 days to the year you can subtract 122 for the one-third of the time you spend in bed and another 45 for the three hours a day used in eating. From the remaining 198 take away 90 for summer vacation and 21 for Christmas and Easter vacations. The days that remain are not even enough to provide for Saturdays and Sundays. Too ancient and obvious a trick to use in serious business, you might say. But the United Automobile Workers insist in their monthly magazine, Ammunition, that it is still being used against them.
The gap between advancing book prices and authors’ earnings, it appears, is due to substantially higher production and material costs. Item: plant and manufacturing expenses alone have risen as much as 10 to 12 per cent over the last decade, materials are up 6 to 9 per cent, selling and advertising expenses have climbed upwards of 10 per cent. Combined boosts add up to a minimum of 33 per cent (for one company) and to nearly 40 per cent for some of the smaller houses. Actually, if each item making up the cost of publishing this book has risen around ten per cent, the total cost must have climbed by about that proportion also. The logic that permits adding those percentage rises together could lead to all sorts of flights of fancy. Buy twenty things today and find that each has gone up five per cent over last year. That “adds up” to one hundred per cent, and the cost of living has doubled. Nonsense.
in the marginYou need to avwrage percentages
How Does He Know?
Ask the question we dealt with in an early chapter: Is the sample large enough to permit any reliable conclusion?
What’s Missing?
Thirty-three and one-third per cent of the women at Hopkins had married faculty members! The raw figures gave a clearer picture. There were three women enrolled at the time, and one of them had married a faculty man.
There is something fascinating about science. One gets such wholesale returns of conjecture out of such a trifling investment of fact.