I'm annoyed by bad science. Poorly designed experiments. Bad survey design. Ignorance of the process of science.
For example, in an upper-level psych class in college, our textbook had an experiment. In an effort to find where thirst was detected in the body, the experimenters injected hypertonic saline solution into the hepatic portal vein (liver) and the jugular vein (neck) to see if thirst was detected in the liver or the brain.
Can you believe that? Someone (I assume a student) designed and proposed an experiment; that was reviewed by the instructor; that was reviewed by at least one committee (ethics). It was performed, written up and submitted for publication. There it was peer-reviewed and published. Then someone decided it was really cool and found its way into a college textbook.
No one, evidently, in that long, involved process knew that veins move blood towards the heart. Away from the organ. All that they measured was which would make the hypertonicity go systemic faster.
Last night I got a call:
"I'd like to ask you a few questions for a survey. What region are you in?"
"You called me. How can you not know what region I'm in?"
"Are you in Washington State?"
"If you knew, why did you ask?"
"What is your opinion on the changes to the health care law?"
"Federal or State?"
"Excuse me?"
"Are you talking about the changes to federal laws or the changes to state laws? They both changed last year, right?"
"I... I don't know."
"How can you not know? You're trying to conduct a poll on a law and you don't know which law?
"The survey form doesn't say."
"Well, since I don't know what you're talking about and neither do you, I don't think I can help you. Good bye."
Okay, it was a little mean and maybe a little funny, but it also was wrong and infuriating. The only valid data possible from this survey is measuring the emotional reaction of people who don't know what they are talking about. Nothing else, because by the nature of it, no one COULD know what they are talking about.
What's your opinion on Blixismaciousness?
But this survey was designed and paid for and someone is going to use the data generated. Use it for what? Not to change the unknown and possibly imaginary health law. At least I don't think so. More likely to tell people what other people think: "80% of the people agree on "X" what is the matter with you?" "Over half the people surveyed in your district don't support your position, Senator." "Our survey indicates that people think this is a serious problem, so we need more funding."
If I hadn't hung up, I probably could have figured out what the survey was designed to elicit (and only really good surveys are neutral). Most people have never even seen a neutral survey. Either by design-- which seems quite common now-- or by subconscious bias in the designer, most surveys and many experiments are targeted at a specific result.
And there's some guilt in this mix of feelings as well. When everyone refuses to play the stupid game it only leaves the ones unable to see the stupid game still playing. Do they then wind up driving policy?
10 comments:
I wold have to agree with you about bad science.
Like the 'scientific' survey on racial disparity in a local newspaper that showed a chart of the city where non-whites were being discriminated against in their choice of low income housing. The bright red outlines of certain 'discriminatory' neighborhoods clearly showed where non-whites are, evidently, not welcome. But when you looked at the tiny numbers (representing complaints, not proven cases of discrimination) on the chart, you saw like - 1, or 2 (in one area of the city, 5 whole people had complained). And, it's a known fact that zombies like brains. I love science, otherwise, how would I know which brand of shave cream to buy?
Or it was a push poll, designed to manipulate your opinion through leading questions...
R E
Preach it! I can't remember the last time I took any survey that was not attempting to lead me in some direction. Poor questions, poor answers to select from, etc. The results being fed to some policy group that will use it to influence policies at various levels of government.
As for lots of the studies out there, pitiful is all I can say.
What's your opinion on Blixismaciousness?
I believe Blixismaciousness is the technique used to trick Mister Mxyzptlk into saying his name backwards thus returning him to the 6th dimension.
And to quote Homer Simpson statistics can be used to prove anything, jeez 79% of everyone knows that.
You weren't mean you just didn't play their game. I'm sure what ever poly-sci intern they had on the phone asked his boss what to do if they call someone who is informed (thats rare)
"When everyone refuses to play the stupid game it only leaves the ones unable to see the stupid game still playing. Do they then wind up driving policy?"
I think that is assumes that you go passive when you stop playing the stupid game.
You can always channel your own personal Statler and Waldorf and opine on the goings-on and the players. :)
actually, a significant number of such calls aren't actually for surveys of any sort. They are poorly veiled attempts to sway opinion.
For instance:
"What's your opinion on Blixismaciousness?"
"Would your opinion change if you knew that it is personally responsible for killing millions of kittens and ducklings?"
"Would your opinion be different if you understood that approving of it will result in having the dead bodies of all those kittens and ducklings delivered to your children's school?"
Cool blog post on polls:
http://statswithcats.wordpress.com/2011/05/08/polls-apart/
Could be a push-poll. Or a poorly-designed one staffed by some kid making minimum wage or getting paid a pittance for each one s/he completes. I've had people who couldn't read the questions out loud call. All they have is a call list and a check-list, they don't know much of anything past that.
I usually find push-polls obvious after two or three questions. Major pollsters generally use better help and have questions that are somewhat better thought out and more directly to their survey point.
The stupid question I've been asked a few times by pollsters is "Do you think the country is headed in the right direction?"
The answer is no, but I haven't believed the country is heading in the right direction under the last four presidents, two of them Republicans and two Democrats. There are any number of directions that I consider not right. What's the point of measuring vague dissatisfaction?
Post a Comment