Study Questions Safety Claims About Tesla Autopilot’s Steering Feature

An analysis of government data disputes a finding by safety regulators and promoted by Tesla Inc. that crash rates of its older vehicles declined after they were equipped with an automatic steering capability.

The study, released Friday by Quality Control Systems Corp., a data analysis company in Maryland, examined the National Highway Traffic Safety Administration’s 2017 conclusion that there was a 40 percent reduction in the rate of crashes by Tesla vehicles that resulted in the deployment of air bags after they were equipped with Autosteer, a part of the automaker’s suite of automated driver-assist systems it markets as Autopilot.

The data that the NHTSA used to reach that conclusion was incomplete and sometimes ignored by the agency in its analysis, according to Quality Control Systems. It attempted to re-create the NHTSA’s analysis using the same data used by the agency, which the agency provided following a public-records lawsuit filed by Quality Control Systems. The company has criticized the NHTSA over auto safety issues in the past.

“Our replication of NHTSA’s analysis of the underlying data shows that the agency’s conclusion is not well-founded,” Randy Whitfield, president of Quality Control Systems, wrote in the report.

In a statement, the NHTSA said it is “reviewing the report released by Quality Control Systems Corp. with interest and will provide comment as appropriate.”

The NHTSA’s original crash-reduction finding garnered broad attention as an apparent validation of Tesla’s Autopilot system from the U.S. government’s auto safety regulator. It was included in a January 2017 agency investigative report on 2014-2016 model year Teslas with Autopilot, the agency’s first such probe of a semi-autonomous driving system.

The inquiry was prompted by a 2016 fatal crash of a Tesla Model S with Autopilot engaged in which the vehicle drove under a tractor-trailer that was crossing a Florida highway, shearing off the Tesla’s roof and killing the driver, a former Navy SEAL. NHTSA ended the inquiry without finding a defect.

Tesla picked up the NHTSA’s finding and cited it in public statements last year, including one after a Model X driver in California died in a crash after his electric SUV drove into a highway barrier while Autopilot was engaged.

The NHTSA has come under under pressure to better explain how it made its finding. Last May, the agency clarified its initial observation, saying the “cursory” comparison of crash rates did not assess the effectiveness of Autosteer or whether crashes occurred while the system was engaged.

The company has updated its Autopilot software several times since the NHTSA first reported the lower rate of crashes. Last year, Tesla began releasing Autopilot crash rates calculated by the company, most recently finding that vehicles in the fourth quarter of 2018 experienced one crash for every 2.9 million miles driven while using Autopilot compared to every 1.6 million miles driven by Teslas not using Autopilot.

In a statement, Tesla said the analysis by Quality Control Systems represents about 1 percent of the miles Tesla vehicles today have driven with Autopilot engaged, and about 0.5 percent of the miles Tesla vehicles have driven overall.

The company also reiterated some of the findings of the NHTSA’s investigation, pointing out that the agency did not identify any defects in the design or performance of Autopilot.

The system uses cameras and sensors to monitor the environment surrounding the car, and can perform lane changes and maintain speed automatically. But the system cannot fully pilot the car on its own, and the company says drivers must remain attentive and keep their hands on the steering wheel.

The Quality Control Systems study, previously reported by the auto-news website The Drive, draws no conclusion of the relative safety benefits of Autopilot but renews questions about NHTSA’s calculation.

“What I really took away was there wasn’t the kind of data submitted to NHTSA that would give an agency the kind of confidence to make the statement it did,” said Bryant Walker Smith, a law professor at the University of South Carolina who studies autonomous vehicle policy. “That’s not scandalous — that’s an agency that’s overworked, overwhelmed, that did a back of the envelope calculation that probably took on greater significance than anyone intended it to.”

Whitfield, the Quality Control Systems president, says the NHTSA relied on only partial mileage data to calculate the rate of air bag-deployment crashes before and after Autosteer was installed.

According to Whitfield’s analysis, complete mileage data both before and after Autosteer installation was found on 5,714 vehicles out of the 43,781 vehicles studied by the NHTSA. Calculating the rate of airbag deployments per million miles driven on that subset of vehicles showed the rate of airbag-deployment crashes actually increased some 59 percent after Autosteer was installed, from 0.76 airbag deployments per million miles before Autosteer installation to to 1.21 per million miles after, the study found.

The report cautioned, though, that questions remain about the value and completeness of the data sought by NHTSA. “We are not in a position to judge the trustworthiness of the data per se,” the report says.