How might we accelerate advent of autonomous driving?
What are the challenges that hold self-driving cars back? That’s the question our ML research led us to. Can it be accelerated?
There are more than a million deaths caused by traffic accidents worldwide every year, 3 parking lots per vehicle are needed in a typical city and typical European car sits over 90 % of time just waiting while 6-15 M US people don’t have access to a vehicle at the same time.
Are we years or decades from autonomy saving millions of lives, changing the way we commute and improving our cities to be designed for humans rather than cars?
Are we there yet?
After the 2018 hype peak and following sobriety, we now see a path towards full autonomy. Yet a bumpy path. But at least leaders in the field now describe achievable challenges to be overcome instead of vague goals like years ago. Pretty tough challenges though. So where are we now?
This summary is made from publicly available information on the web, conferences and presentations and hypotheses just by thinking how we can accelerate the trend towards autonomy. If you consider some info outdated, please suggest updates.
The race of approaches
Waymo started to operate the first public fully autonomous raid-hailing service. Yet the operation is still geo-fenced to the Phoenix area, heavily relies on precise 3D maps with resolution in centimeters and only 5-10 % of rides are fully autonomous without the need for a human operator. This is the result of a 3 years testing with a closed group of passengers and a fleet of 600 cars.
Meanwhile Tesla announced that their autonomous driving architecture is full-self-driving ready and already started rolling it out to first beta customers. (Let’s not confuse readiness with capability though.)
They nicely represent the two distinct approaches, so the bets were made.
Precision & redundancy vs. abundant data
The two approaches have their own advantages and drawbacks.
- Precision & redundancy: Reliable fail-safe limited by area, costs and growth speed:
Waymo counts on reliable lidars providing a full (yet sparse) 3D scene of all surroundings. They rely on accurate 3D maps that needs to be collected in any area they want to operate in. That data is supplemented by further image recognition data from cameras, radars and ultrasonic sensors. While it gives plenty of redundant data to rely on, it also limits how fast they can grow to new areas, thus limiting their growth. - Abundant data limited by a need for driver’s baby-sitting:
Tesla ditched lidars as unnecessary and relies heavily on image processing from vehicle cameras to reconstruct all the scene, supported by radars and ultrasonics.
While it was probably the only affordable approach for Tesla to benefit from their huge fleet (1000× more cars, 200× miles (2B) than Waymo) and drivers teaching the system how to handle risky situations, it also means that no Tesla can drive itself alone without a baby-sitting of a human driver who continues to be responsible for accidents. Nobody knows how long it will take them to get to the point where it will be more reliable than human drivers in any situation, even the hard ones.
What will be the winning approach that will reach cities at scale faster? More data to train autonomous systems? Or reliable redundant fail-safe methods limited by 3D maps?
Challenges, plus tricks of the rest of the pack
What are some challenges to be overcome identified by the creators of autonomous vehicles?
WAYMO principal scientist Drago Anguelov emphasized:
- a long-tail testing challenge
- behavioral corner cases with not enough training data
WAYMO director of engineering Sasha Arnoud claims that:
- when you are 90 % done you have 90 % to go
- long tail of very rare situations of diverse situations including objects states like flashing police cars
Tesla’s AI director Andrej Karpathy emphasizes:
- lots of edge-cases across countries in many variations
- plus weird situations of what a car can encounter on roads
Mobileye (Intel, Fiat Chrysler, BMW) already makes profit from delivering driver-assistance systems (ADAS) to major car makers and creating 3D maps from what their sensors see. Their CEO Amnon Shashua claims they are able to reconstruct a 3D scene from even a single camera. They were actually delivering a first generation of Tesla’s autopilot till 2016 but parted their ways because they insist on their partners not having their own autonomous R&D.
Aurora (VW & Hyundai) CEO Chris Urmson in this interview and talk (from previous job at Google) wants:
- a magic wand to solve perception forecasting what will happen in next seconds
Aurora co-founder Sterling Anderson wants to make autonomy broadly available to all who share the vision and considers tech challenges:
- forecasting intent and future behavior of all actors and how they’ll respond to actions of your own
- long-tail of development
Voyage (Fiat Chrysler) CEO Oliver Cameron thinks to partner with industry is the right way to go (MB, NVIDIA, Uber ATG) and describes:
- hard edge-cases like moving foliage/bushes
- risk of misidentified or missed objects
Their inspiring approach is to focus on retirement communities. While it might be surprising at first, it’s quite a match with calm, distant areas to serve, people with special needs or without an ability to drive and less demands on the speed of transportation. He claims that seniors are not afraid of this technology, because they perceive it rather a more accessible taxi than some high-tech oddity.
APTIV/Motional’s (Hyundai) CEO Karl Iagnemma & director Oscar Beijbom, present challenges:
- completeness and biases of data
- not enough data from rare crashes
Cruise (GM, Honda) CTO Kyle Vogt in his presentation and interview mentions we need to:
- do thousands of little things in handling every variation of each situation, context and intent (acknowledges humans for being excellent in edge-cases)
- maximize diversity of data, large datasets, mix of maneuvers
- focus on disengaging as a signal
- price of datasets and learning: collecting and more importantly labeling 1 M miles costs 1 $B. It takes weeks to train and test a new version.
- benchmarking and estimating performance and categorizing safety risks
Comma.ai founder George Hotz considers a challenge:
- interaction between actors (counterfactuals)
- getting real human behavior
- prediction of occluded areas
Optimus Ride co-founder Sertac Karaman sees challenging to:
- simulate and predict human behavior (what will people do at a specific intersection hundreds of times?)
- typical accidents
Lyft level 5’s Head of Research Peter Ondrúška published about a need to:
- learn from human driving and experiences
- he sees an advantage in acquiring data to feed algorithms with detailed real-world driving scenarios at scale through a rideshare network
Stanford professor, Policies and Safety, Chris Gerdes asks for sharing information about edge-case scenarios to learn collectively like in Aviation Safety Information Analysis and Sharing (ASIAS) system: voluntarily sharing safety information across the industry with anonymized assessment/benchmarking of how they perform in comparison with others.
Will data decide?
What repeats across is the need to train for a very long tail of edge-cases (especially for rare critical events) and predicting real human behavior and interactions. How much of the data is needed so the fleet is ready to go into the wild to outmatch human drivers in cities at scale?
- Will providing in order of magnitude more real street driving clips of the edge-cases help companies reach full autonomy at scale across the world earlier?
- Can the same recorded data be adapted for multiple autonomous systems and positions and characteristics of their sensors?
Now e. g. Tesla has an advantage of its 1000× bigger fleet and 200× more miles street-clips gathered (2 B miles) compared to Waymo and MobilEye claims to be collecting 6 millions km daily from their driving asistents in VWs, BMWs and other cars. And the gap grows.
There seems to be players with and without a direct potential to get abundant data. Like Tesla/MobilEye/Lyft level 5 vs. Waymo/Cruise/Aurora… Will the data be the final winning trick or does it have a lower importance in the whole mix of complexities of autonomous systems? Obviously those with the data are not willing to share.
Is the data what holds us back? What if everyone had enough data, especially about the rare accidents and the only limit would be an ability to use it? How would it accelerate autonomy?
How might we accelerate it?
What if we create consumer products that will collect real-world driving clips and behavior faster? There’s plenty of synergies in collecting data this way: from improving safety and security, evidence for accidents reporting, to monitoring pavement conditions, free parking spots and “real” “real-time” traffic info and more. (Note wejo’s traffic intelligence.)
There is already a first example: nauto collects real-world driving data by offering a driving assistant to improve safety of professional fleets on the road. They claim they collected 800 M miles with risk events identified. So far they partner only with car makers but not the autonomous cars companies. Is there too little “interesting moments” compared to billions of Tesla to make any difference?
Can we provide autonomous car makers an order of magnitude more real-life rare edge-cases data to train their systems cheaper? How could it accelerate maturing the autonomous systems?
Could “ASIAS for autonomous driving” that would allow autonomous vehicles creators to share data of rare edge-cases at high volumes help them the same way its aviation counterpart helps airlines to eliminate even subtle issues and remove operations blind-spots?
Are we there yet?
Although Tesla markets its autopilot to be “full self-driving capable” in the future, as Oliver Cameron pointed out in examples of driver interventions it needed from its driver, it’s clear how many remaining challenges they still have ahead to win the long-tail game. This seems more like some kind of BSD (Babysitted Self Driving) instead of the FSD (Full Self Driving).
Waymo announced no plans to mature out from their Phoenix sandbox into the wild yet.
While MobilEye announced the launch of their AV system in 2022 and demonstrated its capabilities in 40 mins uninterrupted driving video, it is unclear what level of autonomy under what constraints or limitations they can achieve at a wider scale in general conditions across cities.
I don’t dare to bet on who will win the race. Not even if lidar fail-safe or vision only approach will have an edge (depth perception just from cameras seems getting traction as light.co demonstrated, so these might even converge at the end of the day). But my personal guess is that it will take us at least 10 years for autonomous cars to reach major capital cities because it will take a huge amount of time and kilometers ridden to collect data to tame the long-tail of all the variations of the situations that can happen on the road to outperform humans reliably in general conditions without relying on the human interventions.
One diverted shortcut though
There are two areas though, where the autonomy will come sooner than the robo-taxis. Trucks in logistics and a last-mile delivery. Those will make services quicker and cheaper. But those won’t change cities as significantly as autonomous ride-hailing.
Only the ride-hailing will change the way we commute by helping us to get rid of all the hassle of car ownership, making it cheaper, more available to all while reclaiming car parking space cities back to people. And.. saving millions of lives.
Are we there yet?
What is your opinion? What do I miss? How might we accelerate the advent of autonomous vehicles? Share in comments.
