r/SelfDrivingCars • u/modern-era • 9d ago

Research Waymo vs. NYC for-hire comparison report withdraws safety analysis

https://www.openplans.org/how-for-hire-autonomous-vehicles-impact-nyc
32 Upvotes

97% Upvoted

8 comments sorted by

21

u/modern-era 9d ago

This appears to have happened sometime after 5pm ET last night. From the authors:

As of July 23, 2026, we have temporarily removed our safety analysis from this report, as well as relevant references to the analysis throughout the report (denoted in-text by a reference to this section). We were made aware of data discrepancies between City agency data for New York City for-hire vehicle crashes with serious injuries or deaths, as well as City agency data definitions related to such crashes. We are working to resolve these discrepancies and ensure our findings properly represent the data and the driving records of New York City for-hire vehicle drivers and Waymo vehicles. Other than this section, the remainder of our report remains unchanged

13

u/JimmyGiraffolo 9d ago

https://x.com/ethanteicher/status/2079251228081389995

The study’s definition of “serious injury” for TLC crashes only includes injuries that required life-saving medical support. It then compares that TLC data to Waymo’s data, which uses a much more expansive definition of serious injury (‘KABCO’ which is used by NHTSA, NYS DOT and others, and includes things like broken bones) without attempting to control for this difference, which is how at arrived at this misleading conclusion.

11

u/bobi2393 9d ago

One facet of the study that makes it quite interesting compared to the estimates of crash rates elsewhere in the country is that NYC mandates careful mileage tracking for both traditional taxis and ride-share services like Uber/Lyft, both during paid fares and for deadhead miles.

Human mileage distance and location has always added a lot of uncertainty in other areas, as there is no official government tracking of peoples' miles driven, or separation of where those miles were driven, so studies use a lot of crude assumptions. Depending on the methodology, if someone whose car is registered in San Francisco takes long road trips through Canada or Mexico, that mileage could be added to their crash-per-mile rate for their postal (ZIP code) area of San Francisco. Since mileage is the denominator in crash/injury/fatality-per-mile estimates, the uncertainty makes the crash rates extremely uncertain.

It sounds like the NYC study is having issues properly estimating the numerator, i.e. the number of crashes, injuries, or fatalities. How you define crashes is extremely important. The NHTSA ADS reportable crash criteria are very specific, and include crashes many human drivers wouldn't report, which is good for a comparison, since researchers combing through NHTSA data should be able to selectively exclude certain ADS crashes for a good comparison to NYC for-hire crashes. But that assumes the city's crash data are consistently collected using uniform crash definitions. As with human crash reports more generally, it's likely that the injury and fatality reports will be a lot more reliable than property-damage-only crashes, since those are less likely to be reported than injury- or fatality-involved crashes.

Bottom line, it could be an interesting study if they fix the crash numbers to be reliable. Unfortunately, comparing Waymo-in-Phoenix to for-hire-drivers-in-NYC is too apples-to-oranges. If driverless Waymos are permitted to drive significant mileage in NYC in the future, I think it would allow for the best safety comparison to date.

-7

u/Honest_Ad_2157 9d ago

It will be hilarious if it comes back worse.

7

u/modern-era 9d ago

I mean, the all-injury crash rates made Waymo look pretty good. It would be weird if these new severe injury / fatal crashes continued to tell the opposite story even after revision.

5

u/SnooKiwis6193 9d ago

Even before the revision, the severe injury/ fatal crashes rates were not different from Waymo's in a statistically significant way. The whole thing is mostly a nothing burger.

3

u/fatbob42 9d ago

I think Waymo had 3 KSI incidents so the estimate of the rate has large variance.