INVESTIGATING THE IMPACT OF IMAGE PROPERTIES ON OBJECT DETECTION PERFORMANCE IN AUGMENTED REALITY SCENARIOS
Board Location: #120
Discipline: Technology and Engineering
Subcategory: Electrical Engineering
Session: 4
Shivani Karanth - North Carolina State University
Co-Author(s): Tim Scargill, Duke University, Durham, North Carolina
State-of-the-art machine learning models for image classification and object detection still underperform in AR scenarios, due to the properties of real environments and user motion that introduce artifacts into the camera images used as input. In this work, I characterize the differences between the images these models are trained on and images collected in real AR scenarios, how camera image properties relate to prediction performance, and explore techniques for better integration of these models with AR. I began by implementing a characterization pipeline in Python using OpenCV to quantify image properties such as brightness, contrast, entropy, edge strength, and the noise of a set of images. Using this pipeline, I characterized existing object detection datasets (e.g., Coco and RF100), as well as my own image datasets that I collected by capturing videos in conditions representative of realistic AR scenarios. I also measured image similarity in the consecutive frames of video datasets to inform how this relates to object detection results. I then measured image classification and object detection performance by implementing a state-of-the art solution, YOLOv8, along with metrics such as mAP (mean Average Precision) and IoU (intersection over union). Finally, I analyzed the relationships between image properties and object detection results to gain insights into how different conditions impact performance. My work is vital for understanding how to achieve high quality, robust AR experiences that incorporate image classification or object detection, and to improve their deployment for a diverse set of audiences in a wide range of environmental conditions.
Funder Acknowledgement(s): Grand Challenges REU - Duke Pratt School of Engineering
Faculty Advisor: Maria Gorlatova, maria.gorlatova@duke.edu
Role: I did all of the research from start to end, with aid from my mentor, Tim Scargill. All of the work completed is primarily my own.