This week I am incredibly fortunate to be attending the Knight Science Journalism’s Medical Evidence Boot Camp. Today was Day One, with Jennifer Croswell, MD, MPH, of the Agency for Healthcare Research and Quality, and was thoroughly compelling. Breaking down, piece by piece, how medical studies are designed, how to understand how risk is quantified—for example, the very critical difference between absolute risk and relative risk—the meaning of various statistics, and how clinical guidelines are created.
I would like to put all of it down in a blog post because the information is so critical to understanding medicine and being able to parse the constant influx of results and associated clamor for our attention. But alas, as is so often the case, the spirit is willing but the flesh is weak. In other words, it’s almost time for dinner. So, in lieu of a full recall, here’s a tidbit that I loved (among the many tidbits that I loved equally as much).
The P value. The statistic to which we’ve trained our eyes to run when presented with an abstract. Here is something I didn’t know: what the P stands for. Simply, probability. Speaking of probability, probably many of you already knew this. But I didn’t. I always thought, “P means statistically significant” but somehow never managed to connect P with probability. A small matter, but still.
Also, .05, you know the number that we always consider to be the barometer of significance? It’s an arbitrary number, an agreed-upon value that indicates the likelihood that the result is not due to chance.
And, here is my favorite gem: a lower P value does not mean that the findings are more dramatic. Here is where I risk looking foolish! I always thought that if the P value was really, really, really low – like .00000001 – that meant the study findings were somehow better than if the P value was simply .04. Nope! It doesn’t mean that at all. Both of those P values tell you the same thing: that the observed effect was probably not due to chance. (As an example of how P values can be misused, the excellent Jennifer Croswell showed us an advertisement in which a company is touting a P value of .000009 for its recent study.)
Lastly, P values are NOT a measure of the quality of the study. A study could have been put together in a terrible way, but the P value can still be statistically significant. P values are not a mark of trial quality, and they are not a mark of a measure of the importance of the finding. Croswell gave us the imaginary (I think it was made up…) example of a drug that lowers fever in children by .1 degree. The study was statistically significant, but is lowering a fever from 103.5 to 103.4 meaningful? No.
So, hopefully more to come this week, but a little bit from today’s incredible wealth of information and accompany rich and valuable discussion.