ChatGPT flunks self-assessment test for urologists

At a time of growing interest in the potential role of artificial intelligence (AI) technology in medicine and healthcare, a new study reported in Urology Practice finds that the groundbreaking ChatGPT chatbot performs poorly on a major specialty self-assessment tool.

ChatGPT achieved less than a 30% rate of correct answers on the AUA's widely used Self-Assessment Study Program for Urology (SASP). "ChatGPT not only has a low rate of correct answers regarding clinical questions in urologic practice, but also makes certain types of errors that pose a risk of spreading medical misinformation," comment Christopher M. Deibert, MD, MPH, and colleagues of University of Nebraska Medical Center.

Recent advances in large language models (LLMs) provide opportunities for adapting AI technology as a tool for mediating human interaction. "With adequate training and application, these AI systems can process complex information, analyze relationships between ideas, and generate coherent responses to an inquiry," note the authors.

ChatGPT (Chat Generative Pre-Trained Transformer) is an innovative LLM chatbot that has spurred interest in use in a wide range of settings—including health and medicine. In one recent study, ChatGPT scored at or near passing levels on all three steps of the United States Medical Licensing Exam (USMLE), without any special training or feedback on medical topics. Could this innovative AI-trained tool perform similarly well on a more advanced test of clinical knowledge in a surgical specialty?

To find out, Dr. Deibert and colleagues evaluated ChatGPT's performance on the AUA's Self-Assessment Study Program (SASP)—a 150-question practice examination addressing the core curriculum of medical knowledge in urology. The SASP is a valuable test of clinical knowledge for urologists in training and practicing specialists preparing for Board certification. The study excluded 15 questions containing visual information such as pictures or graphs.

Overall, ChatGPT gave correct answers to less than 30% of SASP questions: 28.2% of multiple-choice questions and 26.7% of open-ended questions. The chatbot provided "indeterminate" responses to several questions. On these questions, accuracy was decreased when the LLM model was asked to regenerate its answers.

For most open-ended questions, ChatGPT provided an explanation for the selected answer. The explanations provided by ChatGPT were longer than those provided by SASP, but "frequently redundant and cyclical in nature," according to the authors.

"Overall, ChatGPT often gave vague justifications with broad statements and rarely commented on specifics," Dr. Deibert and colleagues write. Even when given feedback, "ChatGPT continuously reiterated the original explanation despite it being inaccurate."

ChatGPT's poor accuracy on the SASP contrasts with its performance on the USMLE and other graduate-level exams. The authors suggest that while ChatGPT may do well on tests requiring recall of facts, it seems to fall short on questions pertaining to clinical medicine, which require "simultaneous weighing of multiple overlapping facts, situations and outcomes."

"Given that LLMs are limited by their human training, further research is needed to understand their limitations and capabilities across multiple disciplines before it is made available for general use," Dr. Deibert and colleagues conclude. "As is, utilization of ChatGPT in urology has a high likelihood of facilitating medical misinformation for the untrained user."

More information: Linda My Huynh et al, New Artificial Intelligence ChatGPT Performs Poorly on the 2022 Self-assessment Study Program for Urology, Urology Practice (2023). DOI: 10.1097/UPJ.0000000000000406

Citation: ChatGPT flunks self-assessment test for urologists (2023, June 6) retrieved 13 June 2023 from https://medicalxpress.com/news/2023-06-chatgpt-flunks-self-assessment-urologists.html

This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

How To Earn A Public Health Nursing Certification

Editorial Note: We earn a commission from partner links on Forbes Advisor. Commissions do not affect our editors' opinions or evaluations.

Nurses change lives every day in hospitals and clinics. Public health nurses take their practice beyond individual care to create change at a larger level—anything from local vaccine initiatives to national policies.

Public health nurses may provide some direct patient care, but their job also incorporates advocacy and education. They promote and protect their communities’ health using their clinical training and an understanding of social and political sciences.

You don’t need a public health nursing certification to get into this field. But earning a professional distinction can supplement your nursing education and demonstrate your commitment to constant learning.

The only dedicated certification for public health nurses, the Advanced Public Health Nursing Certification, has been retired and is now only available for renewal. However, other relevant certifications can help you build your skills and advance your public health career.

For example, the nationally recognized Certified in Public Health credential doesn’t specifically cover nursing, but it’s available for nurses with public health work experience. Public health nurses with relevant academic histories may qualify to become Certified Health Education Specialists as well.

Find more information about these certifications, how much they cost and their eligibility requirements below.

Public health nurses incorporate social and political sciences into their nursing practice. With a focus on population health rather than individual care, these professionals create and implement programs that promote wellness. They protect community health and work to prevent disease, illness, disability and death.

In general, public health professionals focus on prevention. They create educational programs, conduct research and recommend policies to promote the health of vulnerable populations. Public health nurses also identify and direct care for individuals whose conditions may put others at risk, such as those with infectious diseases.

Public health nurses work in all levels of government—including local, state and federal—and community-focused organizations. Schools, health departments, correctional facilities and community health clinics all employ public health nurses. Public health nurses are well-suited for leadership positions in these settings as well.

The PHNA-BC is the only certification specifically for public health nursing, but it’s been retired by the American Nursing Credentialing Center (ANCC). This credential is only available for already-certified nurses to maintain or renew. Nurses whose certifications expired more than two years ago are no longer eligible for renewal.

Renewal costs $350 for nonmembers of nursing associations, with discounts available for members.

This certification must be renewed every five years. Maintaining certification involves three components: an active nursing license, continued education and professional development activities. All nurses must complete 75 hours of education related to their certification.

You can fulfill the professional development requirement by completing academic credits, giving presentations, publishing peer-reviewed articles or volunteering. Practice hours count toward this requirement, but they are not mandatory for renewal.

The ANCC recommends creating a five-year plan for completing renewal requirements and tracking your progress on its website. When it’s time to renew, any tracked proof of professional development will transfer to your application.

The PHNA-BC may be the only certification dedicated to public health nurses, but don’t worry if you don’t already have it. Other options can showcase your training in public health. Maintaining any relevant certification demonstrates your commitment to continued education, professional growth and ethical practices.

The CPH distinction isn’t specifically for nursing, but it’s a great fit for any public health professional. Created by the National Board of Public Health Examiners, this certification demonstrates both professional and personal skills related to the field. The CPH is recognized as the national standard for public health professionals.

You don’t need a public health degree to earn this certification; nurses can qualify with a bachelor’s in nursing and five years of public health work experience.

CPH exam topics include leadership, communication, law and ethics, policy issues, health equity and social justice. The exam comprises 200 questions with a four-hour time limit. Members of some organizations qualify for discounts, but the regular fee is $385.

Similar to the PHNA-BC, you must complete professional development activities to maintain CPH certification. Renewal costs $95 every two years.

Public health nurses and health education specialists may have different job titles, but their roles overlap. Public health educators promote wellness by collecting data, creating programs, training providers and advocating for policies that support health.

Academic history impacts eligibility for the CHES credential. Health education degree graduates qualify easily. Nurses may be eligible if they have at least a bachelor’s in nursing and 25 semester hours of courses related to health education. Topics include evaluation and research, planning, implementation, advocacy and ethics.

If you have any questions about your eligibility, the National Commission for Health Education Credentialing offers a prescreening service for $25. If you don’t qualify, the commission provides an explanation and guidance for eligibility.

Once approved, you can sit for the 165-question exam. The exam fee ranges from $230 to $400, depending on your student status and registration date.

To become a public health nurse, first earn a nursing degree and pass the NCLEX-RN® exam. You don’t need a public health degree, but public health nurses do need to maintain their nursing license.

Public health nursing focuses on population-level health rather than individual care. Public health nurses work in government and community organizations to conduct research, create and implement programs, and advocate for policy changes that promote health.

Iowa finally passed a law to license midwives. Will it help sway medical field doubters?

desmoinesregister.com cannot provide a good user experience to your browser. To use this site and continue to benefit from our journalism and site features, please upgrade to the latest version of Chrome, Edge, Firefox or Safari.