Sunday, May 18, 2008

Scientifically Validating Experience

I don't have a fever, which feels strange. Soon it will feel normal again, but for right now it's a bit odd. And I'll miss the dreams. Somewhere around 103 degrees the dreams get very interesting.

Epistemology is a really interesting concept and a useful tool to analyze cultures, people, opponents. It is the study of how a group or a person comes to believe that things are 'true'. In some cultures, tradition is truth.  The night ascension of Mohammed or the midnight ride of Paul Revere are simply facts.

In martial arts, the epistemology can get really weird.  People have a need to validate their skills, but are rarely sure how to safely do so.  We can't line up ten thousand slaves and strike each one on a different point and see what happens (though in one kwoon I was told that this was exactly how their  vital points chart was derived- an unnamed Emperor experimented on 10,000 slaves.)  So do you take that story as truth?  If you do, can you then take the conclusions, the list of effects, as truth?  Maybe. Depends on your epistemology.  Can you 'experiment' or 'pressure test' in the ring or the octagon?  Sure, you can experiment anywhere you want.  Doesn't mean the experiment is valid, however.

In our culture, generally, we feel more comfortable if someone in a white coat with a PhD after the name gives us the green light.  Not always- the scientist has to agree with what we already believe or he will be accused of "selling out to big business" or being a "tool of special interest groups" but generally, we like it when scientists confirm what we want to believe.

My degree is in experimental psychology.  Psychology has drifted very far from scientific rigor on a lot of points, but one of the best things about that course of study was some extensive work in designing experiments, analyzing experimental design and the number crunching once the results come in.

So I see some of the martial arts instructors and (especially) entrepreneurs who claim valid and deep scientific backing for what they teach... and it's largely junk science.  Sometimes the experiment didn't conclude what the martial artist claims it did.  Sometimes the design itself, though published, was fundamentally flawed.  Sometimes it was 'entertainment science' like the National Geographic "Fight Science" where apples are compared to oranges.  Often the person using the study never really understood it.

The weirder part is that it doesn't really matter- it's all for comfort anyway.  The guys in white coats won't be anywhere in the area when you need to apply your skills to keep crawling for one more day.  Some of the conclusions work, even when they back them up with junk science.

Yeah, because that's the way this stuff works, that's the way people work.  You talk to enough people, especially combat veterans and you hear things- "I was so scared I shit my pants!" "I couldn't remember how to pull a trigger." "My fingers couldn't work the bolt on my rifle." And you start putting things together.  This scared, this happens.  More scared, more happens.  Sometimes the martial arts instructor/entrepreneur (now a martial researcher! Yeah howdy!) does a study, often with no training in how to do a study or an experiment.  More often they start looking for research that supports their points, probably reading only the abstracts and not able to understand the actual experiment itself. So we have martial research on the effects of fear derived from the research based on running on a treadmill- because the researcher was looking at stress and he found studies based on a cardio stress test.

It doesn't matter because the researcher started with a good conclusion. Fact: the more scared you are the shittier you perform.  That is enough to start improving training.  Getting studies to back it up is largely window dressing.

But it can matter once it is established and starts to work the other way.  When you start extrapolating from junk science or irrelevant, poorly understood experiments and drawing conclusions from those.

One case, not related to MA because every MA case I know has the potential to hurt a friend's feelings:

Our local paper published a study a few years back about hunger in our fair state.  It had shocking and glaring conclusions that one in fifteen people right here were at grave risk of starvation. WTF?  I know some of the poorest people in the largest urban area.  If not for meth, the obesity rate would be astronomical. No one is starving to death. I was raised as a poor kid in the poorest part of the state. No one starved to death.  You would be hard pressed to find anyone who had starved to death, ever in this region who wasn't an elderly shut-in.  So I looked at the study.  Ever waited until a paycheck come in to do your grocery shopping?  That puts you at extreme risk for starvation.  Not for missing a meal, or living out of you cupboards, but starvation!!! Ever smell a restaurant and not go in because you didn't have money with you?  That's grave risk of starvation.  Buddy, you won't make it through the night.

The questionnaire was that poorly designed.  In and of itself it was a joke, just a tool for some folks to get worked up who wanted to get worked up. Relatively harmless.  Except people who either never read or couldn't understand the actual survey started throwing money at a problem that didn't exist.

They feel vindicated.  As far as anyone can name, no one has starved (other than shut-ins) in our fair state since the study came out... of course, no one did before, either.

No comments: