Before Getting Behind the Headlines

The Behind the Headlines project has been going for a while now with aim to tell the truth about stories that are in the news. The Science Media Center (SMC) are starting a new initiative to involve statisticians before a science news story is release to help journalist get the story straight. This is called Before the Headlines.

I tried a first attempt at critiquing a sample paper. I then had a read of some of the other reviews on the SMC webpage to compare. For all of the papers the general responses seem very similar. They talk about the most obvious statistical things to be careful about, as I have. These are:

• What are the assumptions? Are we assuming that things are similar to one another or that something things don't change?
• How are things split-up and why? Are things stratified by age or time or something else?
• What are the confounders? Have they been adjusted for? This is an opportunity to think a little creatively about the problem.
• How big is the data set? Does it take multiple readings?
• Is this hypothesis generation? Is this a case of multiple testing so bound to find something significant.
• Does it just make common sense?? Would people really behave in a certain way? Do the results actually mean anything useful?
• Pull-out some headline statistics. Are these sensible?

Serum Vaccine Antibody Concentrations In Children Exposed to Perfluorinated Compounds JAMA, January 25, 2012—Vol 307, No. 4

Claim supported by evidence?

This paper does claim that there is an association between elevated exposures to PFCs in Faroese children aged 5 and 7 years and reduced humoral immune response to routine childhood immunizations. The paper does not claim that there is a causal relationship It does suggest that this could be the case however.

Summary

The paper does not over-stretch the results of the study by claiming causality [strength] This is for a rather specific cohort in the Faroe Islands and only considers 2 ages (3 measurements) (5 (x2) and 7 years). It says PFC has a half-life of 4 years so is this long enough. Is the study investigating the PFC passed from mother to child or over directly to the child time e.g. 0-5 or 5-7 years old? [limitation]

Study Conclusions

Conclusions are not exaggerated. As ever, more work is needed…

Strengths/Limitations

Well recorded what the details of the study were. Tried to take into account certain covariates and likelihood ratios too. Study uses the fishing community of the Faroe Islands because “frequent intake of marine food is associated with increased exposures to PFCs.” What other characteristics distinguish them from other populations? Perhaps high exposure is a problem but lower exposure is not e.g. does not scale (down) linearly.          The assumption is that all of the mothers and kids are equivalent so that the exposure over time and biological impact and effect are comparable. What about other covariates like profession or diet?          How do these results translate to adult populations?

Glossary

Any specific expertise relevant to studied paper (beyond statistical)?

I am a statistician but do not work as a trials statistician so have mainly applied common sense in this case.