I’m sorry. I’m really, really sorry. The Gary Null thing just won’t leave me alone. I was browsing his site again and ran into this. It’s an example of why some people shouldn’t try to do science…
I’ve written earlier on what science actually is, but some of it bears repeating. In medicine, there are proven ways to study health and disease. In fact, that has been the big revolution in medicine: more important than any single drug or vaccine is the discovery that statistics can be used to identify problems and find out how to fix them.
Some of the earliest statistical work was done by Ignaz Semmelweis in 19th century Europe. He came up with the revolutionary idea that doctors were spreading disease through maternity wards. He tested his hypothesis, kept good data, and used early statistical techniques to find out if he was right or wrong (he was right, but ridiculed and died young in an insane asylum).
Evidence-based medicine is now the standard. No drug, no treatment, no diagnostic test is approved without passing rigorous statistical testing. Of course, given the nature of numbers and people, sometimes we still get it wrong. But that kind of “wrong” is different from the kind of wrong seen here:
Mr Dr. Null presents the “Hair Study”. What kind of study is it?
Ok, I’ll admit that I’m about to bore you to tears. I won’t hold it against you if you skip to a more entertaining post, or to the end of this one. Really. But I promise if you keep reading, you’ll learn something important.
When looking at a health-related study, as this one purports to be, there are a few questions that must be asked, the most important of which are, “What is the purpose”, “How was it done”, and “Do the conclusions match the facts”.
OK, stick with me now…
A) Purpose: “…[to] study of the effects of nutrition, exercise, stress management and environmental adjustments upon the growth of human hair…”
OK, he states the purpose…good. Is it a well-phrased question? Well, not really, but it can still be done…sort of. In general, it is best to lay out your null hypothesis (Oh…my…God! I never even saw the pun approaching. I’m not sure what to do with it yet.) but a general statement of purpose will do.
In order to collect and analyze data, you must know what you are looking for. The statement is a bit too vague, and I would send it back to my student for revision with the following questions:
- What “effects” are you going to study? Hair length? Hair growth rate?
- What operational definitions of your parameters (nutrition, etc) are you planning to come up with?
B) Methods (how is it done):
Participants kept diaries and took notes of any changes in health, e.g. energy levels, sleep patterns, gastrointestinal changes, allergy responses, patterns of colds, flu, infections and sick days. Photos of the hair and scalp were taken and detailed objective and subjective analyses were recorded at the study sessions.
I have a few questions for you before I let you do this study. Here are some of them:
- What type of study are you intending? A randomized controlled trial? A cohort study? It looks most like a cohort study, except that there is an intervention imposed on the subject (exercise, stress management, etc), so maybe it really is a case-control study. I’m not sure.
- How will you choose your subjects?
- Where is your control group?
- How will you assess your data? Using well-developed measures, or “patient diaries”?
- How will you analyze your data? Will you use the standard techniques for a cohort study? Will you calculate Odds Ratios? Relative Risk and Number Needed to Treat? You’ll have to know what kind of study it is and collect the data properly to do that.
C) Conclusions: well, there was no actual statistical analysis presented.
the study concludes that 96% of the participants have a higher level of consistent energy within their waking hours and less sleep required per night. This information is taken from diaries which were kept on a daily basis by the participants. Out of 703 “Hair Study” participants 549 have become controls. These respondents did not adhere to the protocol and reported no changes in hair growth, darkened hair color or being stabilized.
OK, now I’m nauseated. Where did these “conclusions” come from? How does one become a control subject after the fact? Controls are to be chosen before the study begins, not afterwards. You cannot make meaningful statements by looking at a “control” group that was assigned post facto by their own non-compliance.
So what are we to make of this “Hair Study”? What kind of study are you? Why, you’re not a study at all!
There is no useful information to take away here, except that science should be left to people willing to take the time and energy to do it correctly.
