Monday, May 4, 2015

Diving into the data

Every day or so the results of an important study hits the news, and it’s the task of some cube-bound reporter somewhere to interpret and accurately report what it means. Studies are popular news and not easy ones to get right. Sadly, the real importance is often missed and gets misrepresented by journalists. In fact, it’s often missed by everybody, which is shaky since many of these studies are what policymakers rely on.

For one thing, “correlation” and “causation” are incorrectly used as interchangeable and conclusions of preliminary studies are taken as gospel. This is a combination of ignorance concerning the scientific method and the hyper-intellectual jargon that researchers use in their paper, rendering it nearly indecipherable.

Sometimes, a study misses the mark so badly that it raises a red flag to even a TMZ gossip journalist, let alone someone who spends a good part of their day emerged in research material (such as myself).  

The National Transportation Safety Board cranks out its fair share of studies, and most get a lot of attention. Recently, NTSB released a safety report titled “Commercial Vehicle Onboard Video Systems.” The paper attempted to reveal the benefits of video cameras in commercial vehicles as it pertains to safety. The study went on to give recommendations to various transportation organizations based on the findings. Here’s their recommendation that was sent to the American Bus Association, United Motorcoach Association, American Trucking Associations, American Public Transportation Association, National Association for Pupil Transportation, National Association of State Directors of Pupil Transportation Services, and National School Transportation Association:

Encourage your members to ensure that any onboard video system in their vehicles provides visibility of the driver and of each occupant seating location, visibility forward of the vehicle, optimized frame rate, and low-light recording capability.

That recommendation was based on the report’s study that involved two case studies. Two. And the two cases that were used to reach NTSB’s conclusions looked at the cameras of commercial vehicles involved in a crash: a school bus and a motorcoach. The cameras of a truck driver were not represented. Despite the lack of evidence of cameras in trucks, NTSB extended its recommendation to the American Trucking Associations.

Even if a third case study using truck cameras were used in the report, it is still not sufficient evidence to reach any conclusion. Anecdotal evidence has no place in the scientific method unless it is being used to formulate a hypothesis. Beyond that, anecdotes should take a back seat to scientific evidence when drawing a definitive conclusion. Even then, such a study should be peer reviewed and replicated before any confidence can be given. That is not happening here.

Land Line senior editor David Tanner recently reported on a bill that would prohibit the Environmental Protection Agency from using “secret science.” The bill aims to ensure that research is transparent and reproducible before implementing regulations. I recommend that this bill be extended to include all government research, not just the EPA.

Implementing regulations and policies based on anecdotal or otherwise shoddy evidence is both dangerous and irresponsible. Conclusions based on such evidence can easily be inconsistent with the realities of the world. Furthermore, such research does not address the issue of correlation versus causation.

NTSB Senior Biomechanical Engineer Dr. Kristin Poland wrote to me in an email that the Florida crash was chosen since it “was the first case with a continuous video system that captured all phases of the crash.” The Nebraska crash was used “to highlight lessons learned from these video systems.” It’s a good thing my follow-up question was whether or not they plan on using scientific evidence rather than anecdotes. Dr. Poland’s response:

The types of crashes encountered by the NTSB are extremely diverse and therefore a case study methodology is commonly used. Since the prevalence of these systems is continually increasing, our recommendation to NHTSA recommends that NHTSA develop standardized procedures for collecting and using pertinent video recordings in their crash database systems, with appropriate access controls. These databases are commonly used by researchers to study crashes using the scientific method due to the large sample sizes and the ability to compare like crash types.

I agree that the cases are extremely diverse. When anecdotes are diverse, it stands to reason that the conclusions will vary case-by-case. A one-size-fits-all recommendation may not be applicable. I also agree that NHTSA should compile a large database for researchers to use. What I do not agree with is reaching out to transportation organizations with recommendations without the data and subsequent research in place.

If I wanted to, I could find two cases where people wearing flip-flops tripped and injured themselves. From there I could form a hypothesis stating that flip-flops are dangerous and go on to do a scientific experiment proving (or disproving) my theory. What I shouldn’t do is take those two anecdotes and recommend to every shoe manufacturer that they redesign their flip-flops. That is essentially what is going on here.

I understand what Dr. Poland and her colleagues are trying to do, but when it comes to recommendations that can influence policy and the way we live, I want it based on research that can result in more absolute conclusions.