��ɫӰ��

��ɫӰ�� Associate Professor of Electrical and Computer Engineering Joohee Kim and Professor of Electrical and Computer Engineering Ken Choi have been awarded $1.6 million in funding over 4.5 years from HL Mando for a project developing control systems for autonomous vehicles.

Kim will be working on developing artificial intelligence models that will be able to take in information from cameras and a light detection and ranging (LIDAR) sensor and use it to assess road conditions, such as whether the road is wet or dry and detecting the presence of hazards such as ice, potholes, or objects on the road. She will also be using generative vision language models to generate synthetic data that can improve the models where photo data is lacking. 

Choi will be developing computing hardware that is optimized to run the AI models, which is expected to offer an 83 percent decrease in power consumption compared to running the models on more generalized hardware such as a graphic processing unit. 

The AI models will be developed in three stages. First Kim will use a camera-based classification model. This involves taking photos of road surfaces in many conditions and using it to train the model. 

“The objective is to predict the condition of the road and use this to improve the safety of autonomous cars,” she says. 

Next the model will be extended to what is needed in real life. The front-facing camera on an autonomous vehicle sees much of its surroundings beyond just the road and needs to both identify the road and evaluate its condition. This will be done with a camera-based segmentation model, which adds an additional layer of interpretation to the camera-based classification model. 

This model will be based on Kim’s previous work developing real-time segmentation of driving scenes, such as identifying roads, pedestrians, cyclists, cars, and more. 

Finally, Kim aims to address road surface conditions that may not be easily identifiable through a photo. 

“It is quite difficult to identify potholes and bumps using only camera images, especially in bad weather conditions such as rain, storms, ice, or snow,” she says.  

For this, Kim will use data from a LIDAR sensor, which offers more 3D information. These data will be integrated into the overall AI model, making it a multimodal segmentation model. The LIDAR sensor will offer a greater understanding of road surface features identified by the camera-based part of the model.  

Since a foundation of the AI model is interpreting photo data, Kim is also considering how to improve that data set. Most large-scale photo data sets of roads are dominated by photos of dry and sunny road conditions, leaving the most important road conditions that the model needs to learn about in the minority.  

To improve the quality of the AI model prediction, she will utilize generative vision language models to create images of road surfaces under the conditions where photo data is lacking. 

“A lot of research has been done to use generative vision language models to generate more realistic synthetic data so that we can use these types of augmented data sets for various vision applications,” she says. 

Once the AI model is developed, it will need computing hardware to run on. Its complexity means that trying to run it on standard out-of-the-box hardware would be too slow for the needed application. 

“Big vendor companies such as NVIDIA, Intel, Microsoft, and Google provide some general solutions for hardware design, but these are far from the optimal for this specific application,” Choi says.  

He says designing custom hardware that is optimized for the models that Kim develops will improve processing speed and reduce power consumption, making it more useful for application in an actual autonomous vehicle. 

Choi’s focus is on developing hardware that doesn’t require a lot of power to run, offering energy savings. He says the main challenge for this project will be managing the use of multiple sensors, and he plans to achieve this using field-programmable gate arrays, which are integrated circuits with the flexibility to be customized to desired specifications.  

“We want to optimize fully based on the algorithm,” says Choi. 

This project is being conducted in collaboration with 10 other research institutions and companies. 

Image: Graphic of image segmentation using CNNs to develop control systems for autonomous vehicles.