A year and a half ago an Uber self-driving car hit and killed Elaine
Herzberg. I wrote at
the time:
The dashcam video from the Uber crash has been released. It's really
bad. The pedestrian is slowly walking their bike left to right across
a two lane street with streetlights, and manages to get to the right
side of the right lane before being hit. The car doesn't slow down at
all. A human driver would have vision with more dynamic range than
this camera, and it looks to me like they would have seen the
pedestrian about 2s out, time to slow down dramatically even if not
stop entirely. But that doesn't matter here, because this car has
LIDAR, which generates its own light. I'm expecting that when the full
sensor data is released it will be very clear that the system had all
the information it needed to stop in time.
This is the sort of situation where LIDAR should shine, equivalent to
a driver on an open road in broad daylight. That the car took no
action here means things are very wrong with their system. If it were
a company I trusted more than Uber I would say "at least two things
going wrong, like not being able to identify a person pushing a bike
and then not being cautious enough about unknown input" but with Uber
I think they may be just aggressively pushing out immature tech.
On Tuesday
the NTSB released their report (pdf)
and it's clear that the system could easily have avoided this accident
if it had been better designed. Major issues include:
"If we see a problem, wait and hope it goes away". The car was
programmed to, when it determined things were very wrong,
wait one second. Literally. Not even gently apply the brakes. This
is absolutely nuts. If your system has so many false alarms that you
need to include this kind of hack to keep it from acting erratically,
you are not ready to test on public roads.
"If I can't stop in time, why bother?" When the car concluded
emergency braking was needed, and after waiting one second to make
sure it was still needed, it decided not to engage emergency braking
because that wouldn't be sufficient to prevent impact. Since
lower-speed crashes are far more survivable, you definitely still want
to brake hard even if it won't be enough.
"If I'm not sure what it is, how can I remember what it was
doing?" The car wasn't sure whether Herzberg and her bike were a
"Vehicle", "Bicycle", "Unknown", or "Other", and kept switching
between classifications. This shouldn't have been a major issue,
except that with each switch it discarded past observations. Had the
car maintained this history it would have seen that some sort of large
object was progressing across the street on a collision course, and
had plenty of time to stop.
"Only people in crosswalks cross the street." If the car had
correctly classified her as a pedestrian in the middle of the road you
might think it would have expected her to be in the process of
crossing. Except it only thought that for pedestrians in crosswalks;
outside of a crosswalk the car's prior was that any direction was
equally likely.
"The world is black and white." I'm less sure here, but it
sounds like the car computed "most likely" categories for objects, and
then "most likely" paths given their categories and histories, instead
of maintaining some sort of distribution of potential outcomes. If it
had concluded that a pedestrian would probably be out of the
way it would act as if the pedestrian would definitely be out
of the way, even if there was still a 49% chance they wouldn't be.
This is incredibly bad, applying "quick, get it working even if it's
kind of a hack" programming in a field where failure has real
consequences. Self-driving cars have the potential to prevent
hundreds of thousands of deaths a year, but this sort of reckless
approach does not help.
(Disclosure: I work at Google, which is owned by Alphabet, which owns
Waymo, which also operates driverless cars. I'm speaking only for
myself, and don't know anything more than the general public does
about Waymo.)
Content warning: discussion of death
A year and a half ago an Uber self-driving car hit and killed Elaine Herzberg. I wrote at the time:
On Tuesday the NTSB released their report (pdf) and it's clear that the system could easily have avoided this accident if it had been better designed. Major issues include:
"If we see a problem, wait and hope it goes away". The car was programmed to, when it determined things were very wrong, wait one second. Literally. Not even gently apply the brakes. This is absolutely nuts. If your system has so many false alarms that you need to include this kind of hack to keep it from acting erratically, you are not ready to test on public roads.
"If I can't stop in time, why bother?" When the car concluded emergency braking was needed, and after waiting one second to make sure it was still needed, it decided not to engage emergency braking because that wouldn't be sufficient to prevent impact. Since lower-speed crashes are far more survivable, you definitely still want to brake hard even if it won't be enough.
"If I'm not sure what it is, how can I remember what it was doing?" The car wasn't sure whether Herzberg and her bike were a "Vehicle", "Bicycle", "Unknown", or "Other", and kept switching between classifications. This shouldn't have been a major issue, except that with each switch it discarded past observations. Had the car maintained this history it would have seen that some sort of large object was progressing across the street on a collision course, and had plenty of time to stop.
"Only people in crosswalks cross the street." If the car had correctly classified her as a pedestrian in the middle of the road you might think it would have expected her to be in the process of crossing. Except it only thought that for pedestrians in crosswalks; outside of a crosswalk the car's prior was that any direction was equally likely.
"The world is black and white." I'm less sure here, but it sounds like the car computed "most likely" categories for objects, and then "most likely" paths given their categories and histories, instead of maintaining some sort of distribution of potential outcomes. If it had concluded that a pedestrian would probably be out of the way it would act as if the pedestrian would definitely be out of the way, even if there was still a 49% chance they wouldn't be.
This is incredibly bad, applying "quick, get it working even if it's kind of a hack" programming in a field where failure has real consequences. Self-driving cars have the potential to prevent hundreds of thousands of deaths a year, but this sort of reckless approach does not help.
(Disclosure: I work at Google, which is owned by Alphabet, which owns Waymo, which also operates driverless cars. I'm speaking only for myself, and don't know anything more than the general public does about Waymo.)
Comment via: facebook