Evaluating Artificial Intelligence Chatbots for Patient Education in Oral and Maxillofacial Radiology

Document Type

Article

Publication Date

6-2025

Publisher

Elsevier

Source Publication

Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology

Source ISSN

2212-4403

Original Item ID

DOI: 10.1016/j.oooo.2025.01.001

Abstract

Objective

This study aimed to compare the quality and readability of the responses generated by 3 publicly available artificial intelligence (AI) chatbots in answering frequently asked questions (FAQs) related to Oral and Maxillofacial Radiology (OMR) to assess their suitability for patient education.

Study Design

Fifteen OMR-related questions were selected from professional patient information websites. These questions were posed to ChatGPT-3.5 by OpenAI, Gemini 1.5 Pro by Google, and Copilot by Microsoft to generate responses. Three board-certified OMR specialists evaluated the responses regarding scientific adequacy, ease of understanding, and overall reader satisfaction. Readability was assessed using the Flesch-Kincaid Grade Level (FKGL) and Flesch Reading Ease (FRE) scores. The Wilcoxon signed-rank test was conducted to compare the scores assigned by the evaluators to the responses from the chatbots and professional websites. Interevaluator agreement was examined by calculating the Fleiss kappa coefficient.

Results

There were no significant differences between groups in terms of scientific adequacy. In terms of readability, chatbots had overall mean FKGL and FRE scores of 12.97 and 34.11, respectively. Interevaluator agreement level was generally high.

Conclusions

Although chatbots are relatively good at responding to FAQs, validating AI-generated information using input from healthcare professionals can enhance patient care and safety. Readability of the text content in the chatbots and websites requires high reading levels.

Comments

Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology, Vol. 139, No. 6 (June 2025): 750-759. DOI.

Creative Commons License

Creative Commons Attribution 4.0 International License
This work is licensed under a Creative Commons Attribution 4.0 International License.

Link to Full Text

Share

COinS