Evaluation of the Traffic Environment and Risk Factor Understanding Performance of Video-Language Models in Driving Situations: A Comparative Study Based on a Structured 3-Stage Evaluation

Yongsun Lee; Minjee Kim; Jiyoung Woo; Byeongjoon Noh

doi:10.7470/jkst.2026.44.4.722

All Issue

2026 Vol.44, Issue 4 Preview Page

Article

Evaluation of the Traffic Environment and Risk Factor Understanding Performance of Video-Language Models in Driving Situations: A Comparative Study Based on a Structured 3-Stage Evaluation 주행 영상에서 비디오-언어 모델의 주행 환경 및 위험 요인 이해 성능 평가: 구조화된 3단계 벤치마크 기반 비교 연구

31 August 2026. pp. 722-736

PDF XML

Abstract

This study proposes a three-stage structured benchmark for evaluating the ability of video-language models (VLMs) to understand driving environments and risk factors. The proposed framework defines driving risk understanding as a stepwise reasoning process consisting of spatiotemporal environment perception (Q1), inter-agent interaction understanding (Q2), and critical risk identification (Q3). Each stage is implemented as a multiple-choice evaluation item based on reference labels derived from captions in Comprehensive Vision-Language-Action (CoVLA) dataset, enabling the comparison of not only overall model performance but also stage-specific vulnerabilities that arise during the risk understanding process. Using about 500 driving videos and 1,500 questions, this study evaluates seven VLMs under single-/multi-turn dialogue modes. Quantitative evaluation is conducted based on overall accuracy, question-wise accuracy, and performance changes across dialogue modes, while qualitative analysis examines visual grounding, context utilization, and output stability. The experimental results show that most models perform well in the environment perception stage, where they identify road structures and surrounding objects, but exhibit limitations in the Q3 stage, where such information must be connected to critical risk identification. The results also show that the multi-turn setting improves the performance of some models, while allowing initial errors, hallucinations, and output-format instability to propagate into subsequent responses. These findings demonstrate that the proposed benchmark can serve not merely as a tool for ranking models, but as a diagnostic framework for identifying the stages at which VLM-based driving risk understanding remains vulnerable.

Keywords

driving VLMs

single-turn and multi-turn dialogue

traffic risk understanding

traffic safety evaluation

video-language models

본 연구는 비디오-언어 모델(Video-Language Models, VLMs)을 활용하여 주행 환경 및 위험 요인 이해 성능을 평가하기 위한 3단계 구조화 벤치마크를 제안하고 실증적으로 적용한다. 제안한 평가틀은 시공간 환경 인지(Q1), 에이전트 간 상호작용 이해(Q2), 핵심 위험 식별(Q3)로 이어지는 단계적 추론 과정으로 정의한다. 각 단계는 CoVLA(Comprehensive Vision-Language-Action) 데이터셋의 캡션에서 도출한 기준 라벨을 바탕으로 객관식 평가 문항으로 구현하였으며, 이를 통해 모델의 전체 성능뿐 아니라 위험 이해 과정에서 발생하는 단계별 취약성을 비교할 수 있도록 하였다. 본 연구는 주행 영상 약 500개와 1,500개 질문 항목을 활용하여 최근 7개 VLM을 단일/다중 턴 대화 모드에서 평가하였다. 정량 평가는 전체 정확도, 문항별 정확도, 대화 모드 간 변화량 등 을 중심으로 수행하였고, 정성 분석에서는 시각적 근거화, 맥락 활용, 출력 안정성을 검토하였다. 실험 결과, 대부분의 모델은 도로 구조와 주변 객체를 파악하는 환경 인지 단계에의 성능은 우수했으나, 이를 핵심 위험 요인 식별로 연결하는 Q3 단계에서는 한계를 보였다. 또한 다중 턴 조건은 일부 모델의 성능을 향상시켰지만, 동시에 초기 오류, 환각, 출력 형식 불안정성이 후속 응답으로 전파될 수 있음을 보였다. 이러한 결과는 제안한 벤치마크가 단순한 모델 순위 산출을 넘어, VLM의 주행 위험 이해가 어느 단계에서 취약해지는지를 진단하는 평가 도구로 활용될 수 있음을 보여 준다.

키워드

주행 특화 비전언어 모델

단일 · 다중 턴 대화

교통 위험 이해

주행 안전 평가

비디오-언어 모델

References

Ansarinejad M., Gaweesh S. M., Ahmed M. M. (2025), Assessing the Efficacy of Pre-Trained Large Language Models in Analyzing Autonomous Vehicle Field Test Disengagements, Accident Analysis & Prevention, 220, Elsevier, 108178.

10.1016/j.aap.2025.108178

Arai H. et al. (2025), CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Driving, IEEE/CVF Winter Conference on Applications of Computer Vision, IEEE/CVF.

10.1109/WACV61041.2025.00195

Bai S., Cai Y. X., Chen R. Z. et al. (2025), Qwen3-VL Technical Report, CoRR, abs/2511.21631, arXiv.

Cui C., Ma Y. T., Cao X. B., Ye W. Q., Zhou Y. F., Liang K. et al. (2024), A Survey on Multimodal Large Language Models for Autonomous Driving, 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops, IEEE, 958-979.

10.1109/WACVW60836.2024.00106

Fu C. Y., Dai Y. L., Luo Y. H. et al. (2025), Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE/CVF, 24108-24118.

10.1109/CVPR52734.2025.02245

Jeong E., Oh C. H., Lee S. S. (2017), Is Vehicle Automation Enough to Prevent Crashes? Role of Traffic Operations in Automated Driving Environments for Traffic Safety, Accident Analysis & Prevention, 104, Elsevier, 115-124.

10.1016/j.aap.2017.05.002

Kim J. K., Rohrbach A., Darrell T., Canny J., Akata Z. (2018), Textual Explanations for Self-Driving Vehicles, Computer Vision - ECCV 2018, Springer, 577-593.

10.1007/978-3-030-01216-8_35

Koopman P., Wagner M. (2016), Challenges in Autonomous Vehicle Testing and Validation, SAE International Journal of Transportation Safety, 4(1), SAE International, 15-24.

10.4271/2016-01-0128

Laureshyn A., Svensson A., Hyden C. (2010), Evaluation of Traffic Safety, Based on Micro-level Behavioural Data: Theoretical Framework and First Implementation, Accident Analysis & Prevention, 42(6), Elsevier, 1637-1646.

10.1016/j.aap.2010.03.021

Li K., He Y. N., Wang Y. X. et al. (2025), VideoChat: Chat-Centric Video Understanding, Science China Information Sciences, 68, Science China Press, 200102.

10.1007/s11432-024-4321-9

Li K., Wang Y. X., He Y. N. et al. (2024), MVBench: A Comprehensive Multi-modal Video Understanding Benchmark, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE/CVF, 22195-22206.

10.1109/CVPR52733.2024.02095

Ma Y. T., Cui C., Cao X. B., Ye W. Q., Liu P., Lu J. J. et al. (2024), LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programs, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE/CVF, 15141-15151.

10.1109/CVPR52733.2024.01434

Marcu A. M., Chen L. Z., Huenermann J., Karnsund A., Hanotte B., Chidananda P. et al. (2024), LingoQA: Visual Question Answering for Autonomous Driving, Computer Vision - ECCV 2024, Springer, 252-269.

10.1007/978-3-031-72980-5_15

Nie M. J., Peng R., Wang C., Cai X. Y., Han J. S., Xu H. Y., Zhang L. (2024), Reason2Drive: Towards Interpretable and Chain-Based Reasoning for Autonomous Driving, Computer Vision - ECCV 2024, Springer, 292-308.

10.1007/978-3-031-73347-5_17

Papadoulis A., Quddus M., Imprialou M. (2019), Evaluating the Safety Impact of Connected and Autonomous Vehicles on Motorways, Accident Analysis & Prevention, 124, Elsevier, 12-22.

10.1016/j.aap.2018.12.019

Qian T. B., Chen J. Y., Zhuo L., Jiao Y. H., Jiang Y. G. (2024), NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario, Proceedings of the AAAI Conference on Artificial Intelligence, 38(5), AAAI Press, 4542-4550.

10.1609/aaai.v38i5.28253

Sima C., Renz K., Chitta K., Chen L., Zhang H. J., Xie C. G. et al. (2024), DriveLM: Driving with Graph Visual Question Answering, Computer Vision - ECCV 2024, Springer, 256-274.

10.1007/978-3-031-72943-0_15

Sun G. Z., Yang Y. D., Zhuang J. M. et al. (2025), video-SALMONN-o1: Reasoning-Enhanced Audio-Visual Large Language Model, CoRR, abs/2502.11775, arXiv.

Ulbrich S., Menzel T., Reschka A., Schuldt F., Maurer M. (2015), Defining and Substantiating the Terms Scene, Situation, and Scenario for Automated Driving, 2015 IEEE 18th International Conference on Intelligent Transportation Systems, IEEE.

10.1109/ITSC.2015.164

Wang W. Y., Gao Z. W., Gu L. X. et al. (2025), InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency, CoRR, abs/2508.18265, arXiv.

Westhofen L., Neurohr C., Koopmann T., Butz M., Schuett B. U., Utesch F., Kramer B., Gutenkunst C., Boede E. (2023), Criticality Metrics for Automated Driving: A Review and Suitability Analysis of the State of the Art, Archives of Computational Methods in Engineering, Springer.

10.1007/s11831-022-09788-7

Xie S. Z., Kong L. D., Dong Y., Sima C., Zhang W. J., Chen Q. A., Liu Z., Pan L. (2025), Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectives, CoRR, abs/2501.04003, arXiv.

10.1109/ICCV51701.2025.00621

Xue L., Shu M. L., Awadalla A. et al. (2024), xGen-MM (BLIP-3): A Family of Open Large Multimodal Models, CoRR, abs/2408.08872, arXiv.

Yan Z. A., Li X. H., He Y. N. et al. (2025), VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception, Advances in Neural Information Processing Systems, Curran Associates.

10.52202/085713-3976

Z.ai (2026a), GLM-4.6V-Flash Model Card, Hugging Face, https://huggingface.co/zai-org/GLM-4.6V-Flash.

Z.ai (2026b), GLM-V: GLM-4.6V/4.5V/4.1V-Thinking, GitHub Repository, https://github.com/zai-org/GLM-V.

Zhang B. Q., Li K. H., Cheng Z. S. et al. (2025a), VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding, CoRR, abs/2501.13106, arXiv.

Zhang R. X., Wang B. C., Zhang J. X., Bian Z. L., Feng C., Ozbay K. (2025b), When Language and Vision Meet Road Safety: Leveraging Multimodal Large Language Models for Video-Based Traffic Accident Analysis, Accident Analysis & Prevention, 219, Elsevier, 108077.

10.1016/j.aap.2025.108077

Information

Publisher :Korean Society of Transportation
Publisher(Ko) :대한교통학회
Journal Title :Journal of Korean Society of Transportation
Journal Title(Ko) :대한교통학회지
Volume : 44
No :4
Pages :722-736
Received Date : 2026-05-19
Revised Date : 2026-06-01
Accepted Date : 2026-06-08
DOI :https://doi.org/10.7470/jkst.2026.44.4.722

Journal of Korean Society of Transportation ISSN:1229-1366(Print) 2234-4217(Online) 대한교통학회지

All Issue