Assessing the Cognitive and Content Validity of the Driver Licensing Knowledge Test: An LLM-Based Approach

Geongyu 건규; Seonghyun Cho; Jaeyeong Lee

doi:10.7470/jkst.2026.44.2.232

All Issue

2026 Vol.44, Issue 2 Preview Page Next Page

Article

Assessing the Cognitive and Content Validity of the Driver Licensing Knowledge Test: An LLM-Based Approach LLM 기반 문항 분류를 활용한 운전면허 학과시험의 인지·내용 구조 및 타당성 평가

30 April 2026. pp. 232-249

PDF XML

Abstract

The driver’s license written test is a key assessment system designed to foster safe traffic culture and to select competent novice drivers. However, it has been continuously criticized for failing to sufficiently reflect the cognitive judgment and reasoning processes required in actual driving. Previous studies have mainly focused on macroscopic indicators such as pass rates or test difficulty, or on linguistic bias within test items, without empirically examining the structural relationship between the required cognitive level and examinees’ performance. To address this gap, this study analyzes 989 test-bank items using an LLM-assisted classification combined with expert review, categorizing each item by SOLO (Structure of Observed Learning Outcomes) cognitive level and driving-competence content area, and evaluating performance differences based on item-level accuracy. Results show that more than half of all items fall into the Multistructural level, with very few at the Extended Abstract level, indicating limited coverage of higher-order cognitive processes. Content areas are similarly imbalanced, with heavy concentration in rules/signs/signals and relatively few items assessing hazard perception or situational judgment. Accuracy was highest for relational items, suggesting that some items may be answered through safety-oriented intuition rather than the intended reasoning process. ANOVA confirmed significant performance differences across cognitive levels and content areas, while Cohen’s h revealed minimal age-group differences in basic knowledge items but noticeable performance declines among older adults on relational and hazard-perception items. These findings indicate that age-related differences in cognitive performance were observed in certain cognitive levels and content areas, but that the overall structure of the test—due to its concentration in a limited range of cognitive levels and content domains—has structural limitations in comprehensively assessing the complex reasoning and hazard-detection skills required for safe driving. Accordingly, the results highlight the need for improving the test framework to better reflect age-related cognitive characteristics and for expanding the inclusion of items that demand higher-order cognitive abilities.

Keywords

cognitive complexity

driver’s license written test

driving competence assessment

Large Language Model (LLM)

SOLO taxonomy

운전면허 학과시험은 안전한 교통문화를 형성하고 우수한 신규 운전자를 양성하기 위한 핵심 평가 제도이지만, 실제 운전 수행에서 요구되는 인지적 판단과 사고 과정을 충분히 반영하지 못한다는 지적이 지속되어 왔다. 기존 연구들은 주로 시험의 합격률이나 난이도와 같은 거시적 지표, 혹은 문항의 언어적 편향성에 초점을 맞추었으나, 문항이 요구하는 인지 수준과 응시자 성취 간의 구조적 관계를 실증적으로 규명하지는 못했다는 한계를 가진다. 이에 본 연구는 989개의 운전면허 학과시험 문제은행 문항을 LLM 기반 자동 분류와 전문가 검증을 결합하여 SOLO (Structure of Observed Learning Outcomes) 분류체계와 운전 역량 내용 영역에 따라 문항의 인지·내용 구조를 체계적으로 분류하고, 정답률을 활용해 수행 차이를 분석하였다. 분석 결과, 전체 문항의 과반이 다구조 수준에 집중되고 확장된 추상 문항은 극히 적어 고차 사고 요구가 제한적이었다. 내용 영역 역시 법규·표지·신호 중심으로 편중되어 위험인지·상황판단 문항의 비중이 낮았다. 정답률은 관계적 문항에서 가장 높게 나타나 일부 문항이 의도된 사고 과정보다는 안전지향적 직관에 의해 풀릴 가능성이 확인되었다. ANOVA 분석 결과, 인지 수준과 내용 영역에 따른 수행 차이는 유의하였으며, Cohen’s h 분석에서는 단순 지식형 문항에서는 세대 간 차이가 거의 없었으나, 관계적 사고와 위험인지·상황판단 영역에서는 고령층의 수행 저하가 두드러졌다. 이러한 결과는 일부 인지 수준과 내용 영역에서 연령 증가에 따른 인지 수행 차이가 분석을 통해 확인되었음에도 불구하고, 문항 구성이 특정 인지 수준과 내용 영역에 편중됨으로써 실제 운전 수행에 요구되는 복합적 사고력과 위험 인지 능력을 전반적으로 평가하기에는 구조적 한계를 지니고 있음을 시사한다. 따라서 향후에는 연령별 인지 특성을 반영한 평가체계의 개선과 더불어, 적절한 난이도의 고차 인지 능력을 요구하는 문항의 확대가 필요함을 시사한다.

키워드

인지 복잡성

운전면허 학과시험

운전 역량 평가

대규모 언어모델(LLM)

SOLO 분류체계

References

Baik S. Y. (2018), A Study on the Driver's License Written Test System to Improve New Drivers’ Driving Ability, The Korean Association of Police Science Review, 20(2), 29-65.

10.24055/kaps.20.2.2

Biggs J. B., Collis K. F. (1982), Evaluating the Quality of Learning: The SOLO(Structure of the Observed Learning Outcome) Taxonomy, Academic Press.

Brabrand C., Dahl B. (2009). Using the SOLO Taxonomy to Analyze Competence Progression of University Science Curricula. Higher Education, 58(4), 531-549.

10.1007/s10734-009-9210-4

Cohen J. (2013), Statistical Power Analysis for the Behavioral Sciences, Routledge.

10.4324/9780203771587

Curry A. E., Metzger K. B., Williams A. F., Tefft B. C. (2017). Comparison of Older and Younger Novice Driver Crash Rates: Informing the Need for Extended Graduated Driver Licensing Restrictions. Accident Analysis & Prevention, 108, 66-73.

10.1016/j.aap.2017.08.015

Depestele S., Ross V., Verstraelen S., Brijs K., Brijs T., van Dun K., Meesen R. (2020). The Impact of Cognitive Functioning on Driving Performance of Older Persons in Comparison to Younger Age Groups: A Systematic Review. Transportation research part F: traffic psychology and behaviour, 73, 433-452.

10.1016/j.trf.2020.07.009

European Commission (2006), Directive 2006/126/EC on driving licences (Recast), Official Journal of the European Union, L403, 18-60.

Fisher D. L., Strayer D. L. (2014), Modeling Situation Awareness and Crash Risk, Ann Adv Automot Med, 58, 33.

Fraade-Blanar L. A. et al. (2018), Cognitive Decline and Older Driver Crash Risk, Journal of the American Geriatrics Society, 66(6), Wiley, 1075-1081.

10.1111/jgs.15378

Habibzadeh Y., Yarmohammadian M. H., Sadeghi-Bazargani H. (2023), Driving Hazard Perception Components: A Systematic Review and Meta-Analysis, Bulletin of Emergency & Trauma, 11(1), 1.

Hatakka M., Keskinen E., Gregersen N. P., Glad A., Hernetkoski K. (2002), From Control of the Vehicle to Personal Self-Control; Broadening the Perspectives to Driver Education, Transportation Research Part F: Traffic Psychology and Behaviour, 5(3), 201-215.

10.1016/S1369-8478(02)00018-9

Henriksson W., Sundström A., Wiberg M. (2004), The Swedish driving-license test: A summary of studies from the department of educational measurement, Umeå University. Department of Educational Measurement, Umeå University.

Hodges L. C., Harvey L. C. (2003). Evaluation of Student Learning in Organic Chemistry Using the SOLO Taxonomy. Journal of Chemical Education, 80(7), 785.

10.1021/ed080p785

Horswill M. S., Hill A., Wetton M. (2015), Can a Video-Based Hazard Perception Test Used for Driver Licensing Predict Crash Involvement? Accident Analysis & Prevention, 82, 213-219.

10.1016/j.aap.2015.05.019

Kim H., Lee J. (2025), Large Language Model-Based Prompt Framework for Analyzing Driver’s License Written Test Items, International Journal of Highway Engineering, 27(3), 79-90.

10.7855/IJHE.2025.27.3.079

Krathwohl D. R. (2002), A Revision of Bloom’s Taxonomy: An Overview, Theory Into Practice, 41(4), Taylor & Francis, 212-218.

10.1207/s15430421tip4104_2

Landis J. R., Koch G. G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1), 159-174.

10.2307/2529310

Lee C. M. (2019), A Study on the Traffic Safety Effects and Development Strategies according to the Improvement of Driver's License Test on year of 2016, Master's Thesis, The Graduate School, Ajou University. http://www.riss.kr/link?id=T15094785

Lee C. M., Lee C. K., Lee S. S., Yun I. S. (2020), Study on the Traffic Safety Effects of Driving Test Improvement, The Journal of The Korea Institute of Intelligent Transport Systems, 19(5), 15-28.

10.12815/kits.2020.19.5.15

Lee D. (2020), An Analysis of Sport Pedagogy in Secondary Teacher Certification Examination: Focusing on Sport Pedagogy Since 2014, Korean Society For The Study Of Physical Education, 25(2), 17-34.

10.15831/JKSSPE.2020.25.2.17

Lee J. K., Lee S. W. (2018), A Study on Driver’s License System and Its Improvement, The Korean Association of Police Science Review, 20(5), 233-260.

10.24055/kaps.20.5.9

Lee J., Oh E., Sohn E. H., Kim J. Y., Lee M. Y., Kim J. H., Ko Y. K., Lee A. Y. (2018), Correlation of Driving-related Cognition and Cognitive Impairment in Drivers over the Age of 60, Journal of the Korean Neurological Association, 36(2), 74-80.

10.17340/jkna.2018.2.2

Lee J., Son J. (2025), A Natural Language Processing-Based Analysis of Semantic Bias and Assessment Fairness in Driver's License Written Examinations, The Korean Association of Police Science Review, 27(2), 67-90.

10.24055/kaps.27.2.3

Lee S. (2019), A Study on the Estimating License Demerit Point about the Items Causing Traffic Accidents, J. Korean Soc. Transp., 37(5), 365-374.

10.7470/jkst.2019.37.5.365

McKnight A. J., McKnight A. S. (2003), Young Novice Drivers: Careless or Clueless?, Accident Analysis and Prevention, 35(6), 921-925.

10.1016/S0001-4575(02)00100-8

Salthouse T. A. (2019), Trajectories of Normal Cognitive Aging, Psychology and Aging, 34(1), 17.

10.1037/pag0000288

Stefan, D. A., Heikoop, D. D., de Winter, J. C., & Houwing, S. (2025). Predictors of long-term knowledge retention in the driver theory test. Scientific Reports, 15(1), 20948.

10.1038/s41598-025-04796-6

Thompson E., Luxton-Reilly A., Whalley J. L., Hu M., Robbins P. (2008), Bloom's Taxonomy for CS Assessment, In Proceedings of the Tenth Conference on Australasian Computing Education, 78, 155-161.

Yang S. (2023), Analysis and Improvement of Grammar Questions in the Korean Language Teacher Appointment Examination, The Journal of Learner-Centered Curriculum and Instruction, 23(21), 295-306.

10.22251/jlcci.2023.23.21.295

Yoo K. R., Jo J. (2020), Did Ease on the Drivers’ License Test in 2011 Increase Car Accidents?, Journal of the Korean Data And Information Science Society, 31(1), 149-157.

10.7465/jkdi.2020.31.1.149

Information

Publisher :Korean Society of Transportation
Publisher(Ko) :대한교통학회
Journal Title :Journal of Korean Society of Transportation
Journal Title(Ko) :대한교통학회지
Volume : 44
No :2
Pages :232-249
Received Date : 2025-10-30
Revised Date : 2025-11-25
Accepted Date : 2025-12-29
DOI :https://doi.org/10.7470/jkst.2026.44.2.232

Journal of Korean Society of Transportation ISSN:1229-1366(Print) 2234-4217(Online) 대한교통학회지

All Issue