Publications

Conferences & Workshops

1. 

   Challenges of Data-Driven Simulation of Diverse and Consistent Human Driving Behaviors

   Kalle Kujanpää, Daulet Baimukashev, Shibei Zhu, Shoaib Azam, Farzeen Munir, Gokhan Alcan, and Ville Kyrki

   In , 2024

   Building simulation environments for developing and testing autonomous vehicles necessitates that the simulators accurately model the statistical realism of the real-world environment, including the interaction with other vehicles driven by human drivers. To address this requirement, an accurate human behavior model is essential to incorporate the diversity and consistency of human driving behavior. We propose a mathematical framework for designing a data-driven simulation model that simulates human driving behavior more realistically than the currently used physics-based simulation models. Experiments conducted using the NGSIM dataset validate our hypothesis regarding the necessity of considering the complexity, diversity, and consistency of human driving behavior when aiming to develop realistic simulators.

   @inproceedings{kujanpaa2024challenges,
     show_bib = {true},
     title = {Challenges of Data-Driven Simulation of Diverse and Consistent Human Driving Behaviors},
     author = {Kujanp{\"a}{\"a}, Kalle and Baimukashev, Daulet and Zhu, Shibei and Azam, Shoaib and Munir, Farzeen and Alcan, Gokhan and Kyrki, Ville},
     journal = {arXiv preprint arXiv:2401.03236},
     year = {2024}
   }

2. 

   Exploring Large Language Models for Trajectory Prediction: A Technical Perspective

   Farzeen Munir, Tsvetomila Mihaylova, Shoaib Azam, Tomasz Piotr Kucner, and Ville Kyrki

   In Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, 2024

   Large Language Models (LLMs) have been recently proposed for trajectory prediction in autonomous driving, where they potentially can provide explainable reasoning capability about driving situations. Most studies use versions of the OpenAI GPT, while there are open-source alternatives which have not been evaluated in this context. In this report1, we study their trajectory prediction performance as well as their ability to reason about the situation. Our results indicate that open-source alternatives are feasible for trajectory prediction. However, their ability to describe situations and reason about potential consequences of actions appears limited, and warrants future research.

   @inproceedings{10.1145/3610978.3640625,
     show_bib = {true},
     author = {Munir, Farzeen and Mihaylova, Tsvetomila and Azam, Shoaib and Kucner, Tomasz Piotr and Kyrki, Ville},
     title = {Exploring Large Language Models for Trajectory Prediction: A Technical Perspective},
     year = {2024},
     isbn = {9798400703232},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     url = {https://doi.org/10.1145/3610978.3640625},
     doi = {10.1145/3610978.3640625},
     booktitle = {Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction},
     pages = {774–778},
     numpages = {5},
     keywords = {autonomous driving, large language models, trajectory prediction},
     location = {<conf-loc>, <city>Boulder</city>, <state>CO</state>, <country>USA</country>, </conf-loc>},
     series = {HRI '24}
   }

3. 

   Radar-Lidar Fusion for Object Detection by Designing Effective Convolution Networks

   Farzeen Munir, Shoaib Azam, Tomasz Kucner, Ville Kyrkil, and Moongu Jeon

   In 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), 2023

   Object detection is a core component of perception systems, providing the ego vehicle with information about its surroundings to ensure safe route planning. While cameras and Lidar have significantly advanced perception systems, their performance can be limited in adverse weather conditions. In contrast, millimeter-wave technology enables radars to function effectively in such conditions. However, relying solely on radar for building a perception system doesn’t fully capture the environment due to the data’s sparse nature. To address this, sensor fusion strategies have been introduced. We propose a dual-branch framework to integrate radar and Lidar data for enhanced object detection. The primary branch focuses on extracting radar features, while the auxiliary branch extracts Lidar features. These are then combined using additive attention. Subsequently, the integrated features are processed through a novel Parallel Forked Structure (PFS) to manage scale variations. A region proposal head is then utilized for object detection. We evaluated the effectiveness of our proposed method on the Radiate dataset using COCO metrics. The results show that it surpasses state-of-the-art methods by 1.89% and 2.61% in favorable and adverse weather conditions, respectively. This underscores the value of radar-Lidar fusion in achieving precise object detection and localization, especially in challenging weather conditions.

   @inproceedings{munir2023radar,
     show_bib = {true},
     title = {Radar-Lidar Fusion for Object Detection by Designing Effective Convolution Networks},
     author = {Munir, Farzeen and Azam, Shoaib and Kucner, Tomasz and Kyrkil, Ville and Jeon, Moongu},
     booktitle = {2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC)},
     pages = {3575--3582},
     year = {2023},
     organization = {IEEE}
   }

4. 

   Channel boosting feature ensemble for radar-based object detection

   Shoaib Azam, Farzeen Munir, and Moongu Jeon

   In 2021 IEEE Intelligent Vehicles Symposium (IV), 2021

   Autonomous vehicles are conceived to provide safe and secure services by validating the safety standards as indicated by SOTIF-ISO/PAS-21448 (Safety of the intended functionality). Keeping in this context, the perception of the environment plays an instrumental role in conjunction with localization, planning and control modules. As a pivotal algorithm in the perception stack, object detection provides extensive insights into the autonomous vehicle’s surroundings. Camera and Lidar are extensively utilized for object detection among different sensor modalities, but these exteroceptive sensors have limitations in resolution and adverse weather conditions. In this work, radar-based object detection is explored provides a counterpart sensor modality to be deployed and used in adverse weather conditions. The radar gives complex data; for this purpose, a channel boosting feature ensemble method with transformer encoder-decoder network is proposed. The object detection task using radar is formulated as a set prediction problem and evaluated on the publicly available dataset [1] in both good and good-bad weather conditions. The proposed method’s efficacy is extensively evaluated using the COCO evaluation metric, and the best-proposed model surpasses its state-of-the-art counterpart method by 12.55% and 12.48% in both good and good-bad weather conditions.

   @inproceedings{azam2021channel,
     show_bib = {true},
     title = {Channel boosting feature ensemble for radar-based object detection},
     author = {Azam, Shoaib and Munir, Farzeen and Jeon, Moongu},
     booktitle = {2021 IEEE Intelligent Vehicles Symposium (IV)},
     pages = {762--769},
     year = {2021},
     organization = {IEEE}
   }

5. 

   Sstn: Self-supervised domain adaptation thermal object detection for autonomous driving

   Farzeen Munir, Shoaib Azam, and Moongu Jeon

   In 2021 IEEE/RSJ international conference on intelligent robots and systems (IROS), 2021

   The perception of the environment plays a decisive role in the safe and secure operation of autonomous vehicles. The perception of the surrounding is way similar to human vision. The human’s brain perceives the environment by utilizing different sensory channels and develop a view-invariant representation model. In this context, different exteroceptive sensors like cameras, Lidar, are deployed on the autonomous vehicle to perceive the environment. These sensors have illustrated their benefit in the visible spectrum domain yet in the adverse weather conditions; for instance, they have limited operational capability at night, leading to fatal accidents. This work explores thermal object detection to model a view-invariant model representation by employing the self-supervised contrastive learning approach. We have proposed a deep neural network Self Supervised Thermal Network (SSTN) for learning the feature embedding to maximize the information between visible and infrared spectrum domain by contrastive learning. Later, these learned feature representations are employed for thermal object detection using a multi-scale encoder-decoder transformer network. The proposed method is extensively evaluated on the two publicly available datasets: the FLIR-ADAS dataset and the KAIST Multi-Spectral dataset. The experimental results illustrate the efficacy of the proposed method.

   @inproceedings{munir2021sstn,
     show_bib = {true},
     title = {Sstn: Self-supervised domain adaptation thermal object detection for autonomous driving},
     author = {Munir, Farzeen and Azam, Shoaib and Jeon, Moongu},
     booktitle = {2021 IEEE/RSJ international conference on intelligent robots and systems (IROS)},
     pages = {206--213},
     year = {2021},
     organization = {IEEE}
   }

6. 

   ARTSEG: Employing attention for thermal images semantic segmentation

   Farzeen Munir, Shoaib Azam, Unse Fatima, and Moongu Jeon

   In Asian Conference on Pattern Recognition, 2021

   The research advancements have made the neural network algorithms deployed in the autonomous vehicle to perceive the surrounding. The standard exteroceptive sensors that are utilized for the perception of the environment are cameras and Lidar. Therefore, the neural network algorithms developed using these exteroceptive sensors have provided the necessary solution for the autonomous vehicle’s perception. One major drawback of these exteroceptive sensors is their operability in adverse weather conditions, for instance, low illumination and night conditions. The useability and affordability of thermal cameras in the sensor suite of the autonomous vehicle provide the necessary improvement in the autonomous vehicle’s perception in adverse weather conditions. The semantics of the environment benefits the robust perception, which can be achieved by segmenting different objects in the scene. In this work, we have employed the thermal camera for semantic segmentation. We have designed an attention-based Recurrent Convolution Network (RCNN) encoder-decoder architecture named ARTSeg for thermal semantic segmentation. The main contribution of this work is the design of encoder-decoder architecture, which employ units of RCNN for each encoder and decoder block. Furthermore, additive attention is employed in the decoder module to retain high-resolution features and improve the localization of features. The efficacy of the proposed method is evaluated on the available public dataset, showing better performance with other state-of-the-art methods in mean intersection over union (IoU).

   @inproceedings{munir2021artseg,
     show_bib = {true},
     title = {ARTSEG: Employing attention for thermal images semantic segmentation},
     author = {Munir, Farzeen and Azam, Shoaib and Fatima, Unse and Jeon, Moongu},
     booktitle = {Asian Conference on Pattern Recognition},
     pages = {366--378},
     year = {2021},
     organization = {Springer International Publishing Cham}
   }

7. 

   Dynamic Control System Design for Autonomous Car

   Shoaib Azam, Farzeen Munir, and Moongu Jeon

   In VEHITS, 2020

   The autonomous vehicle requires higher standards of safety to maneuver in a complex environment. We focus on control of the self-driving vehicle that includes the longitudinal and lateral dynamics of the vehicle. In this work, we have developed a customized controller for our KIA Soul self-driving car. The customized controller implements the PID control for throttle, brake, and steering so that the vehicle follows the desired velocity profile, which enables a comfortable and safe ride. Besides, we have also catered the lateral dynamic model with two approaches: pure pursuit and model predictive control. An extensive analysis is performed between pure pursuit and its adversary model predictive control for the efficacy of the lateral model.

   @inproceedings{azam2020dynamic,
     show_bib = {true},
     title = {Dynamic Control System Design for Autonomous Car},
     author = {Azam, Shoaib and Munir, Farzeen and Jeon, Moongu},
     booktitle = {VEHITS},
     pages = {456--463},
     year = {2020}
   }

8. 

   Multiple objects tracking using radar for autonomous driving

   Muhamamd Ishfaq Hussain, Shoaib Azam, Farzeen Munir, Zafran Khan, and Moongu Jeon

   In 2020 IEEE International IOT, Electronics and Mechatronics Conference (IEMTRONICS), 2020

   Object detection and tracking are the integral elements for the perception of the spatio-temporal environment. The availability and affordability of camera and lidar as the leading sensor modalities have used for object detection and tracking in research. The usage of deep learning algorithms for the object detection and tracking using camera and lidar have illustrated the promising results, but these sensor modalities are prone to weather conditions, have sparse data and spatial resolution problems. In this work, we explore the problem of detecting distant objects and tracking using radar. For the efficacy of our proposed work, extensive experimentation in different traffic scenario are performed by using our self-driving car test-bed.

   @inproceedings{hussain2020multiple,
     show_bib = {true},
     title = {Multiple objects tracking using radar for autonomous driving},
     author = {Hussain, Muhamamd Ishfaq and Azam, Shoaib and Munir, Farzeen and Khan, Zafran and Jeon, Moongu},
     booktitle = {2020 IEEE International IOT, Electronics and Mechatronics Conference (IEMTRONICS)},
     pages = {1--4},
     year = {2020},
     organization = {IEEE}
   }

9. 

   Visuomotor Steering angle Prediction in Dynamic Perception Environment for Autonomous Vehicle

   Farzeen Munir, Shoaib Azam, and Moongu Jeon

   In 2020 IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia), 2020

   Visuomotor coordination in driving assists the driver in performing necessary action expeditiously. Over the recent years, the development towards autonomous vehicles has accelerated tremendously, and an increase in the computational capabilities motivates us to train the complex deep neural network models using visual cues for better understanding of visuomotor coordination in the autonomous vehicles. In this work, we exploit this problem by using visual cues in the images to predict the steering angle for a self-driving car. The future frames are predicted to estimate a look-ahead steering angle so that self-driving car can make appropriate decisions. In our dynamic predictive model, we have used the deep convolution-LSTM model for predicting the future frames and a series of ResNet blocks for predicting the steering angle for the self-driving car. Moreover, we have used segmentation as an auxiliary information for training the dynamic predictive network. The efficacy of our dynamic predictive model is rigorously tested on our collected and Udacity dataset.

   @inproceedings{munir2020visuomotor,
     show_bib = {true},
     title = {Visuomotor Steering angle Prediction in Dynamic Perception Environment for Autonomous Vehicle},
     author = {Munir, Farzeen and Azam, Shoaib and Jeon, Moongu},
     booktitle = {2020 IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia)},
     pages = {1--6},
     year = {2020},
     organization = {IEEE}
   }

10. 

    Data fusion of lidar and thermal camera for autonomous driving

    Shoaib Azam, Farzeen Munir, Ahmad Muqeem Sheri, YeongMin Ko, Ishfaq Hussain, and Moongu Jeon

    In Applied Industrial Optics: Spectroscopy, Imaging and Metrology, 2019

    The adverse environmental conditions build a bottleneck for the autonomous driving. This challenge is resolved by data fusion of sensor modalities. Here, thermal and Lidar data are fused together for the precise perception of environment.

    @inproceedings{azam2019data,
      show_bib = {true},
      title = {Data fusion of lidar and thermal camera for autonomous driving},
      author = {Azam, Shoaib and Munir, Farzeen and Sheri, Ahmad Muqeem and Ko, YeongMin and Hussain, Ishfaq and Jeon, Moongu},
      booktitle = {Applied Industrial Optics: Spectroscopy, Imaging and Metrology},
      pages = {T2A--5},
      year = {2019},
      organization = {Optica Publishing Group}
    }

11. 

    Automated Taxi Booking Operations for Autonomous Vehicles

    Linh Van Ma, Shoaib Azam, Farzeen Munir, Moongu Jeon, and Jinho Choi

    In 2019 13th International Conference on Signal Processing and Communication Systems (ICSPCS), 2019

    In a conventional taxi booking system, all taxi operations are mostly done by a decision made by drivers which is hard to implement in unmanned vehicles. To address this challenge, we introduce a taxi booking system which assists autonomous vehicles to pick up customers. The system can allocate an autonomous vehicle (AV) as well as plan service trips for a customer request. We use our own AV to serve a customer who uses a mobile application to make his taxi request. Apart from customer and AV, we build a server to monitor customers and AVs. It also supports inter-communication between a customer and an AV once AV decided to pick up a customer.

    @inproceedings{van2019automated,
      show_bib = {true},
      title = {Automated Taxi Booking Operations for Autonomous Vehicles},
      author = {Van Ma, Linh and Azam, Shoaib and Munir, Farzeen and Jeon, Moongu and Choi, Jinho},
      booktitle = {2019 13th International Conference on Signal Processing and Communication Systems (ICSPCS)},
      pages = {1--5},
      year = {2019},
      organization = {IEEE}
    }

12. 

    Where Am I: Localization and 3D Maps for Autonomous Vehicles

    Farzeen Munir, Shoaib Azam, Ahmad Muqeem Sheri, YeongMin Ko, and Moongu Jeon

    In VEHITS, 2019

    The nuts and bolts of autonomous driving find its root in devising the localization strategy. Lidar as one of the newest technologies developed in the recent years, provides rich information about the environment in the form of point cloud data which can be used for localization. In this paper, we discuss a localization approach which generates a 3D map from Lidar’s point cloud data using Normal Distribution Transform (NDT) mapping. We use our own dataset collected using our self driving car KIA Soul EV equipped with Lidar and cameras. Once the 3D map has been generated, we have used NDT matching for localizing the self driving car.

    @inproceedings{munir2019localization,
      show_bib = {true},
      title = {Where Am I: Localization and 3D Maps for Autonomous Vehicles},
      author = {Munir, Farzeen and Azam, Shoaib and Sheri, Ahmad Muqeem and Ko, YeongMin and Jeon, Moongu},
      booktitle = {VEHITS},
      volume = {2019},
      pages = {452--457},
      year = {2019}
    }

13. 

    Object modeling from 3d point cloud data for self-driving vehicles

    Shoaib Azam, Farzeen Munir, Aasim Rafique, YeongMin Ko, Ahmad Muqeem Sheri, and Moongu Jeon

    In 2018 IEEE Intelligent Vehicles Symposium (IV), 2018

    For autonomous vehicles to be deployed and used practically, many problems are still needed to be solved. One of them we are interested in is to make use of a cheap LIDAR for robust object modelling with 3D point cloud data. Self-driving vehicles require accurate information about the surrounding environments to decide the next course of actions. 3D point cloud data obtained from LIDAR give more accurate distance than the counterpart stereo images. As LIDAR generates lowresolution data, the object detection and modeling is prone to produce errors. In this work, we propose the use of multiple frames of LIDAR data in an urban environment to construct a comprehensive model of the object. We assume the use of LIDAR on a moving platform and the results are almost equal to the 3D CAD model representation of the object.

    @inproceedings{azam2018object,
      show_bib = {true},
      title = {Object modeling from 3d point cloud data for self-driving vehicles},
      author = {Azam, Shoaib and Munir, Farzeen and Rafique, Aasim and Ko, YeongMin and Sheri, Ahmad Muqeem and Jeon, Moongu},
      booktitle = {2018 IEEE Intelligent Vehicles Symposium (IV)},
      pages = {409--414},
      year = {2018},
      organization = {IEEE}
    }

14. 

    Autonomous vehicle: The architecture aspect of self driving car

    Farzeen Munir, Shoaib Azam, Muhammad Ishfaq Hussain, Ahmed Muqeem Sheri, and Moongu Jeon

    In Proceedings of the 2018 International Conference on Sensors, Signal and Image Processing, 2018

    Self-driving cars have received a lot of attention in recent years and many stakeholders like Google, Uber, Tesla, and so forth have invested a lot in this area and developed their own autonomous driving car platforms. The challenge to make an autonomous car is not only the stringent performance but also the safety of the passengers and pedestrians. Even with the development of technologies, autonomous driving is still an active research area and still requires a lot of experimentations and making architecture entirely autonomous. The intriguing area of self-driving car motivates us to build an autonomous driving platform. In this paper, we discuss the architecture of the self-driving car and its software components that include localization, detection, motion planning and mission planning. We also highlight the hardware modules that are responsible for controlling the car. The autonomous driving is running state-of-the-art algorithms used in localization, detection, mission and motion planning.

    @inproceedings{munir2018autonomous,
      show_bib = {true},
      title = {Autonomous vehicle: The architecture aspect of self driving car},
      author = {Munir, Farzeen and Azam, Shoaib and Hussain, Muhammad Ishfaq and Sheri, Ahmed Muqeem and Jeon, Moongu},
      booktitle = {Proceedings of the 2018 International Conference on Sensors, Signal and Image Processing},
      pages = {1--5},
      year = {2018}
    }

15. 

    A Benchmark of Computational Models of Saliency to Predict Human Fixations in Videos.

    Shoaib Azam, Syed Omer Gilani, Moongu Jeon, Rehan Yousaf, and Jeong-Bae Kim

    In VISIGRAPP (4: VISAPP), 2016

    In many applications of computer graphics and design, robotics and computer vision, there is always a need to predict where human looks in the scene. However this is still a challenging task that how human visual system certainly works. A number of computational models have been designed using different approaches to estimate the human visual system. Most of these models have been tested on images and performance is calculated on this basis. A benchmark is made using images to see the immediate comparison between the models. Apart from that there is no benchmark on videos, to alleviate this problem we have a created a benchmark of six computational models implemented on 12 videos which have been viewed by 15 observers in a free viewing task. Further a weighted theory (both manual and automatic) is designed and implemented on videos using these six models which improved Area under the ROC. We have found that Graph Based Visual Saliency (GBVS) and Random Centre Surround Models have outperformed the other models.

    @inproceedings{azam2016benchmark,
      show_bib = {true},
      title = {A Benchmark of Computational Models of Saliency to Predict Human Fixations in Videos.},
      author = {Azam, Shoaib and Gilani, Syed Omer and Jeon, Moongu and Yousaf, Rehan and Kim, Jeong-Bae},
      booktitle = {VISIGRAPP (4: VISAPP)},
      pages = {134--142},
      year = {2016}
    }

16. 

    Vehicle pose detection using region based convolutional neural network

    Shoaib Azam, Aasim Rafique, and Moongu Jeon

    In 2016 International Conference on Control, Automation and Information Sciences (ICCAIS), 2016

    In recent years, category-level object detection has gained a lot of attention. In addition to object localization, estimation of the object pose has practical applications in intelligent transportation, autonomous driving and robotics. Parts based models have been used for pose estimation in recent years, but these models depend on manual supervision or require a complex algorithm to locate the object parts. In this work, we have used Convolutional Neural Network for the pose estimation of vehicle in an image. The advantage of multiple classifications of objects at the same time motivates us to choose the convolutional neural network. We make use of state-of-the-art implementation of convolution neural network named the Region Based Convolutional Neural Network(FASTER-RCNN) for estimating the pose of vehicle. We annotate the comprehensive cars dataset of Stanford, required for training the model and upon testing we have achieved good results with good accuracy.

    @inproceedings{azam2016vehicle,
      show_bib = {true},
      title = {Vehicle pose detection using region based convolutional neural network},
      author = {Azam, Shoaib and Rafique, Aasim and Jeon, Moongu},
      booktitle = {2016 International Conference on Control, Automation and Information Sciences (ICCAIS)},
      pages = {194--198},
      year = {2016},
      organization = {IEEE}
    }

17. 

    Face-deidentification in images using restricted boltzmann machines

    M Aasim Rafique, M Shoaib Azam, Moongu Jeon, and Sangwook Lee

    In 2016 11th International Conference for Internet Technology and Secured Transactions (ICITST), 2016

    In this work, we discuss utility of Restricted Boltzmann Machine (RBM) in face-deidentification challenge. GRBM is a generative modeling technique and its unsupervised learning provides vantage of using raw faces data. Faces are deidentified by reconstructed face images from the trained GRBM model. The reconstructed image uses random information from the stochastic units which makes it hard to re-identify from the deidentified face. Experiments show the proposed technique maintain emotions in the test face, which is intrinsic to the modeling capacity of RBM.

    @inproceedings{rafique2016face,
      show_bib = {true},
      title = {Face-deidentification in images using restricted boltzmann machines},
      author = {Rafique, M Aasim and Azam, M Shoaib and Jeon, Moongu and Lee, Sangwook},
      booktitle = {2016 11th International Conference for Internet Technology and Secured Transactions (ICITST)},
      pages = {69--73},
      year = {2016},
      organization = {IEEE}
    }

18. 

    Single object tracking system using fast compressive tracking

    Abdullah Tahir, Shoaib Azam, Sujani Sagabala, Moongu Jeon, and Ryu Jeha

    In 2016 IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia), 2016

    In this work we focused on the application aspect of object tracking for pan-tilt-zoom (PTZ) camera using ordinary webcam mounted on custom made motor-assembly and found that our system is not only robust to illumination conditions but also cost-effective in comparison with PTZ cameras. For object tracking we utilized Fast Compressive Tracking (FCT) algorithm because of its attractive features e.g. online learning, fast computation and robust performance. A PC program interfaced with embedded system through serial RS232 commands motors, hence camera, to real time track desired object in world such that object being tracked remains in the center of image.

    @inproceedings{tahir2016single,
      show_bib = {true},
      title = {Single object tracking system using fast compressive tracking},
      author = {Tahir, Abdullah and Azam, Shoaib and Sagabala, Sujani and Jeon, Moongu and Jeha, Ryu},
      booktitle = {2016 IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia)},
      pages = {1--3},
      year = {2016},
      organization = {IEEE}
    }