As ever, the twitter backlash against the Australian Vaccination Network continues. One particular item has grabbed my attention. @nocompulsoryvac, a twitter account run by Meryl Dorey on behalf of the AVN, tweeted this little gem today, on a claimed 155% risk increase for Autism.
Well, this piqued my interest. That's a fairly spectacular figure, representing a significant 2.5x increase in risk. Wow. And it's quite a specific number too. Must be conducted in quite a rigorous manner, eh?
Well, no. Not really.
You see, even though Meryl left that little gem unattributed*, I decided I was going to hunt down this data point and see if it was really all it was cracked up to be.
So off I went to my favourite search engine, and typed in "155% autism risk vaccine". And what do you know? I found the "study", first result out of the gate.
It turns out that the rigorous, statistically controlled, epidemiological study in question was... a phone survey. By a marketing company. Funded, to the tune of US$200,000 by... Generation Rescue, a partisan antivax organisation founded by a ditzy celebrity and dedicated to proving the MMR/Autism link by any means possible.
Oh I'm sorry, did I say all that? I meant concerned parents' group honestly investigating the MMR/Autism link.
DING-A-LING-A-FUCKING-LING
So aside from the immediate issue with how a poorly designed phone survey somehow gets conflated with real medical research, there are other issues. It seems there was no medical follow-up to confirm diagnoses, no control for known risk factors such as past familial neurological trouble, no reporting on exactly what the poll questions were, and finally no review of the data by any other group - all of which raise serious flags about the reliability of said "study"
But do you know what the best part was?
No, really? Do you want to know?
Well, it seems Meryl can't even report her own partisan studies correctly. The actual results are summarised in the article thusly:
-- Among more than 9,000 boys age 4-17, vaccinated boys were 2.5 times (155 percent) more likely to have neurological disorders, 224 percent more likely to have attention deficit hyperactivity disorder, and 61 percent more likely to have autism.
-- For older vaccinated boys in the 11-17 age bracket, the results were even more pronounced. Vaccinated boys were 158 percent more likely to have a neurological disorder, 317 percent more likely to have ADHD, and 112 percent more likely to have autism.
I've highlighted where Meryl fucked up, not because I think that you, dear reader, are a bit thick, because you're not. You're smart and good looking, and probably get far more casual sex than the average person. I highlighted in in case Meryl comes by, because, you see, she needs help. Meryl's target figure (autism) was 61%. 155% referred to all 'neurological disorders', including the rather fluffily defined ADHD.
Now, it's been a while since I did Statistics 101, but the numbers also seem a little skewed, and oddly reported. The two age groups quoted, for example, are actually just one group, with a subset pulled out to highlight the intended result. What we have is 4-17, and the subset 11-17. What are the numbers for the 4-11 group? We're not told, but we can extrapolate, and someone clever with Maths could probably tease out the numbers. Yes, folks, the 4-11 numbers WOULD be lower, significantly lower, than either of the two groups we quoted. There's no way around that. If your overall sample set has a 61% rate, but one half has a 112% rate, what does the unreported group have? That's right, an even lower rate**. So MMR (and other vaccines) are administered around school starting age, and then very little happens until the age of 11. That's stretching post-hoc ergo propter hoc a bit far, even for Meryl.
Secondly, what we're quoting here is a percentage increase in a rate that itself is not reported. Dr Ben Goldacre devotes an antire chapter of "Bad Science" to "Bad Stats", and this is precisely where he starts out. The number best suited to this kind of reporting is the 'Natural Frequency', that is a number such as "1 in 5000 people". However it's far more sensational, if your study gives you small results, to report the 'relative risk increase'. Let's just break this into another paragraph to make this a little easier, and just make up a hypothetical study of 10,000 subjects
- Background rate of event: 1/10,000, or 100 per million
- Reported rate for study: 3/10,000, or 300 per million
- Relative Risk Increase: 300%
Fuck! 300%! Call the cops! Call the fucking coastguard! Do something! That's a 300% fucking increase!!! Aaaaaaagh!
No, wait. That's an increase of only two reported cases per 10,000 results. We just managed to find two more people than we expected by chance. Out of 10,000 people. That's almost certain to happen if you run more than one study.
This is entirely statistically insignificant. But it's a 300% increase. That's even more then Generation Rescue found in their study. Holy crap, I'm onto something!
Not only that, given that the incredible publicity being generated around the purported vaccine/autism link, I would positively expect misreported results in this kind of survey. A kid that misbehaves a little, in the mind of a mother exposed to too much Jenny on Oprah, is suddenly an ADHD case. A socially awkward kid is suddenly autistic. You can see where I'm going here.
In fact, given the methodology and the background surrounding this specific manufactured 'controversy', I'm really, really surprised the numbers weren't even more skewed.
So, in summary, Meryl is lying again. SURPRISE!
* If there's one thing antivaxers are good at, it's making assertions without attribution. For a period on the AVN's FaceBook page, I made it my business to correctly attribute quotes lifted wholesale by AVN supporters.
** Math geeks, I'm not sure I got the calculations correct, but is it possible there would be a negative result in the lower group?
posted @ Monday, August 17, 2009 5:37 PM