“Experimental validation of camera-based maritime collision avoidance for autonomous urban passenger ferries”
Authors: Øystein Kaarstad Helgesen, Emil H. Thyri, Edmund Brekke, Annette Stahl and Morten Breivik,Affiliation: NTNU, Department of Engineering Cybernetics and Zeabuz AS
Reference: 2023, Vol 44, No 2, pp. 55-68.
Keywords: Maritime autonomy, target tracking, collision avoidance, daylight cameras, full-scale experiments, autonomous surface vehicle, autonomous urban passenger ferries
Abstract: Maritime collision avoidance systems rely on accurate state estimates of other objects in the environment from a tracking system. Traditionally, this understanding is generated using one or more active sensors such as radars and lidars. Imaging sensors such as daylight cameras have recently become a popular addition to these sensor suites due to their low cost and high resolution. However, most tracking systems still rely exclusively on active sensors or a fusion of active and passive sensors. In this work, we present a complete collision avoidance system relying solely on camera tracking. The viability of this autonomous navigation system is verified through a real-world, closed-loop collision avoidance experiment with a single target in Trondheim, Norway in December 2022. Accurate tracking was established in all scenarios and the collision avoidance system took appropriate actions to avoid collisions.
PDF (3291 Kb)
DOI: 10.4173/mic.2023.2.2DOI forward links to this article:
| [1] Mathias Thoresen Paasche, Oystein Kaarstad Helgesen and Edmund Forland Brekke (2023), doi:10.1088/1742-6596/2618/1/012009 |
[1] Bitar, G., Eriksen, B.-O.H., Lekkas, A.M., and Breivik, M. (2019). Energy-Optimized Hybrid Collision Avoidance for ASVs, In 18th European Control Conference (ECC). pages 2522--2529, 2019. doi:10.23919/ECC.2019.8795645
[2] Bochkovskiy, A., Wang, C., and Liao, H.M. (2020). YOLOv4: Optimal Speed and Accuracy of Object Detection, CoRR. abs/2004.10934. https://arxiv.org/abs/2004.10934.
[3] Cormack, D., Schlangen, I., Hopgood, J.R., and Clark, D.E. (2020). Joint Registration and Fusion of an Infrared Camera and Scanning Radar in a Maritime Context, IEEE Transactions on Aerospace and Electronic Systems. 56(2):1357--1369. doi:10.1109/TAES.2019.2929974
[4] Eriksen, B.-O.H. (2019). Collision avoidance and motion control for autonomous surface vehicles , Ph.D. thesis, Norwegian University of Science and Technology. https://ntnuopen.ntnu.no/ntnu-xmlui/handle/11250/2616394.
[5] Eriksen, B.-O.H., Breivik, M., Wilthil, E.F., Flåten, A.L., and Brekke, E.F. (2019). The branching-course model predictive control algorithm for maritime collision avoidance, Journal of Field Robotics. 36(7):1222--1249. doi:https://doi.org/10.1002/rob.21900
[6] Fossen, T. (2021). Handbook of Marine Craft Hydrodynamics and Motion Control, Wiley. https://books.google.no/books?id=tCQqEAAAQBAJ.
[7] Fowdur, J.S., Baum, M., and Heymann, F. (2021). Real-World Marine Radar Datasets for Evaluating Target Tracking Methods, Sensors. 21(14):4641. doi:https://doi.org/10.3390/s21144641
[8] Gaglione, D., Braca, P., and Soldi, G. (2018). Belief Propagation Based AIS/Radar Data Fusion for Multi - Target Tracking, In 21st International Conference on Information Fusion (FUSION). pages 2143--2150. doi:10.23919/ICIF.2018.8455217
[9] Helgesen, O.K., Stahl, A., and Brekke, E.F. (2023). Maritime tracking with georeferenced multi-camera fusion, IEEE Access. 11:30340--30359. doi:10.1109/ACCESS.2023.3261556
[10] Helgesen, O.K., Vasstein, K., Brekke, E.F., and Stahl, A. (2022). Heterogeneous multi-sensor tracking for an autonomous surface vehicle in a littoral environment, Ocean Engineering. 252:111168. doi:https://doi.org/10.1016/j.oceaneng.2022.111168
[11] Kuwata, Y., Wolf, M.T., Zarzhitsky, D., and Huntsberger, T.L. (2014). Safe Maritime Autonomous Navigation with COLREGS, Using Velocity Obstacles, IEEE Journal of Oceanic Engineering. 39(1):110--119. doi:10.1109/JOE.2013.2254214
[12] Kuznietsov, Y., Stückler, J., and Leibe, B. (2017). Semi-Supervised Deep Learning for Monocular Depth Map Prediction, In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pages 2215--2223. doi:10.1109/CVPR.2017.238
[13] Musicki, D., Evans, R., and Stankovic, S. (1992). Integrated probabilistic data association (IPDA), In 31st IEEE Conference on Decision and Control. pages 3796--3798 vol.4. doi:10.1109/CDC.1992.370951
[14] Schuster, M., Blaich, M., and Reuter, J. (2014). Collision Avoidance for Vessels using a Low-Cost Radar Sensor, 19th IFAC World Congress. 47(3):9673--9678. doi:https://doi.org/10.3182/20140824-6-ZA-1003.01872
[15] Schöller, F., Blanke, M., Plenge-Feidenhans’, M., and Nalpantidis, L. (2020). Vision-based Object Tracking in Marine Environments using Features from Neural Network Detections, 21th IFAC World Congress. 53(2):14517--14523. 21st IFAC World Congress. doi:https://doi.org/10.1016/j.ifacol.2020.12.1455
[16] Thyri, E.H., Breivik, M., and Lekkas, A.M. (2020). A Path-Velocity Decomposition Approach to Collision Avoidance for Autonomous Passenger Ferries in Confined Waters, 21th IFAC World Congress. 53(2):14628--14635. doi:https://doi.org/10.1016/j.ifacol.2020.12.1472
[17] Wilthil, E.F., Flåten, A.L., and Brekke, E.F. (2017). A target tracking system for ASV collision avoidance based on the PDAF, In P.Fossen and Nijmeijer, editors, Sensing and Control for Autonomous Vehicles, volume 474, pages 269--288. Springer, Alesund, Norway, 2017. http://dx.doi.org/10.1007/978-3-319-55372-6_13.
[18] Wolf, M.T., Assad, C., Kuwata, Y., Howard, A., Aghazarian, H., Zhu, D., Lu, T., Trebi-Ollennu, A., and Huntsberger, T. (2010). 360-degree visual detection and target tracking on an autonomous surface vehicle, Journal of Field Robotics. 27(6):819--833. doi:https://doi.org/10.1002/rob.20371
[19] Zhang, Z. (2000). A flexible new technique for camera calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence. 22(11):1330--1334. doi:10.1109/34.888718
BibTeX:
@article{MIC-2023-2-2,
title={{Experimental validation of camera-based maritime collision avoidance for autonomous urban passenger ferries}},
author={Helgesen, Øystein Kaarstad and Thyri, Emil H. and Brekke, Edmund and Stahl, Annette and Breivik, Morten},
journal={Modeling, Identification and Control},
volume={44},
number={2},
pages={55--68},
year={2023},
doi={10.4173/mic.2023.2.2},
publisher={Norwegian Society of Automatic Control}
};