Can chatGPT pass a radiology board-style examination?

In a recent study published in the Radiology Journal, researchers performed a prospective exploratory analysis to assess the performance of artificial intelligence (AI)-based ChatGPT on radiology board–style examination questions between February 25 and March 3, 2023.

Study: Performance of ChatGPT on a Radiology Board-style Exam: Insights into Current Strengths and Limitations. Image Credit: MMDCreative/Shutterstock.com

ChatGPT, based on GPT-3.5, is a general large language model (LLM) pre-trained on >45 terabytes of textual data using deep neural networks.

Though not trained in medical data, ChatGPT has shown immense potential in medical data writing and education. Accordingly, physicians are already using ChatGPT with search engines to search for medical information.

ChatGPT is under investigation for its potential use in simplifying radiology reports and aiding clinical decision-making. Additionally, it could help educate radiology students, perform differential and computer-aided diagnoses, and in disease classification.

ChatGPT recognizes relationships and patterns between words across its enormous training data to generate human-like responses.

Though it could generate a factually incorrect response; however, so far, ChatGPT has performed exceptionally well on several professional examinations, e.g., the U.S. Medical Licensing Exam, without any domain-specific pretraining.

Though ChatGPT appears promising for applications in diagnostic radiology, including image analysis, the ChatGPT performance in the radiology domain remains unknown.

More importantly, radiologists must know the strengths and limitations of ChatGPT to use it confidently.

In the present study, researchers included 150 multiple-choice questions with one correct and three wrong answers, which matched the content, style, and difficulty level of the Canadian Royal College examination in diagnostic radiology and the American Board of Radiology Core and Certifying examinations.

These board examinations comprehensively assess conceptual knowledge of radiology and the ability to reason and make a clinical judgment(s).

Two board-certified radiologists independently reviewed these questions and ensured these questions matched specific criteria, e.g., questions did not have images, wrong answers were plausible and similar in length to the correct answer, etc.

At least 10% of questions originated from nine topics listed by the Canadian Royal College to ensure these multiple-choice questions were on topics that comprehensively covered the concept of radiology.

Two other board-certified radiologists classified those 150 multiple-choice questions by type using Bloom Taxonomy principles into lower-order or higher-order thinking.

The team entered all questions with their answer choices into ChatGPT to simulate real-world use and recorded all ChatGPT responses. The Royal College considers ≥70% on all written components as passing scores.

Another two board-certified radiologists subjectively assessed the language of each ChatGPT response for its level of confidence on a Likert scale on a one-to-four, where a score of four indicated high confidence and zero indicated no confidence.

Finally, the researchers also made qualitative observations of the behavior of ChatGPT when they prompted the model with the correct answer.

First, the researchers computed the overall performance of ChatGPT. Next, they compared its performance using Fisher exact test between question types and topics, e.g., related to physics or clinical type.

In addition, they performed subgroup analysis for subclassifications of higher-order thinking questions. The team had subclassified higher-order thinking questions into four groups, involving the description of imaging, clinical management, application of concepts, and disease associations.

Lastly, they used the Mann-Whitney U test to compare the confidence level of responses between correct and incorrect ChatGPT responses, where p-values less than 0.05 indicated a significant difference.

ChatGPT nearly passed radiology board–style examination questions without images in this study and scored 69%.

The model performance was better on questions requiring lower-order thinking involving knowledge recall and basic understanding than those requiring higher-order thinking (84% vs. 60%).

However, it performed well on higher-order questions related to clinical management (89%), likely because a large amount of disease-specific patient-facing data is available on the Internet.

It struggled with higher-order questions involving the description of imaging results, calculation and classification, and application of concepts.

Also, ChatGPT performed poorly on physics questions relative to clinical questions (40% vs. 73%). ChatGPT used confident language consistently, even when incorrect (100%).

The tendency of ChatGPT to produce incorrect human-like responses with confidence is particularly dangerous if it is the sole source of information. This behavior limits the applicability of ChatGPT in medical education at present.

ChatGPT excelled on questions assessing basic knowledge and understanding of radiology, and without radiology-specific pretraining, it nearly passed (scored 69%) a radiology board–style examination without images.

However, radiologists must exercise caution and remain aware of the limitations of ChatGPT, including its tendency to present incorrect responses with 100% confidence. In other words, study findings do not support relying on ChatGPT for practice or education.

With future advancements in LLMs, the availability of applications built on LLMs with radiology-specific pretraining will increase. Overall, the study results are encouraging for the potential of LLMs-based models like ChatGPT in radiology.

Himachal mulls bringing back annual exam system for state board-run schools

Himachal Pradesh School Education Board is contemplating doing away with the term examination system and following a system akin to the Central Board of School Exam (CBSE), with teacher organisations batting for its abolition.

Besides, teacher organisations have attributed the poor results to the term. The same comes in the backdrop of an unusually high number of students facing compartments. This year, the overall pass percentage has also witnessed a significant 14% drop, going down from 93.9% to 79.4%.

The State Teachers Association, Scheduled Castes and Scheduled Tribes Employees Welfare Association have since submitted a memorandum to the board secretary in regards with the proposed change.

Teacher organisations say the system was not well-suited for the geographical location of Himachal that impacts the school calendar, resulting in a failure to allow time for students’ all-round development.

Besides, they noted that two separate term exams for Class 10 and 12 Boards put a financial burden on the children and their parents as they end up paying the examination fee twice in addition to a certain retention fee.

In such a situation, the government is now expected to take a decision on the issue soon.

Teachers bat for annual system

Speaking of the issue, Himachal Rajkiya Prakashak Sangh state president Chittaranjan Kalta said the annual system worked well for most schools in the state and the same reflected in the results too before batting for its reintroduction.

As per the ASER-2022 Himachal Pradesh Rural report, the enrolment of children across all age categories in public schools was significantly higher than in their private counterparts. In government schools, enrolment was highest among girls between the 15-16 age group (82.4%) and boys (77.6%). In the same age range, 3.4% of the children are not enrolled in any school at all.

Notably, Himachal Pradesh had an 82.8% literacy rate — 89.53% male and 75.93% female — per the Census 2011, 8.8% higher than the national average of 74%. The figures were far better than the Census 2001 averages of 85.35% for men, 67.42 for women, and 76.48% overall.

The National Sample Survey Office’s (NSSO) 75th round of surveys on “Household Social Consumption: Education in 2017-18” provides the latest figures. According to the same, in 2017, the research predicted an 86.6% literacy rate for the state, putting the men at 92.9% and women at 80.5%.

All state school’s Pupil Teacher Ratios (PTRs) were compared to those in Haryana, Uttrakhand, Punjab, and the rest of India. When comparing 2020-21 and 2021-22, the state’s secondary school PTR had improved from 7 to 6. When compared to its surrounding states of Punjab, Haryana, and Uttrakhand, as well as the rest of India, Himachal Pradesh scores well on all measures.

The state has also given special focus to the education sector, with the state having 10,758 primary, 1,965 middle, 962 high and 1,999 senior secondary schools in the public sector. The government also runs 166 degree colleges

New India Literacy Programme (NILP), a centrally sponsored scheme for adult education, has also been launched for the 2022-23 to 2026-27 period, with an objective to educate around 89,000 adult illiterate population across the state.

ABOUT THE AUTHOR

Gaurav Bisht heads Hindustan Times’ Himachal bureau. He covers politics in the hill state and other issues concerning the masses. ...view detail