Exploration of Digital Health Technologies

Current Issue:
3articles Vol.2(2024) , Issue 6

Sort by

Latest

Latest

Most Viewed

Most Downloaded

Most Cited

Open Access

Original Article

Assessing the accuracy and readability of ChatGPT-4 and Gemini in answering oral cancer queries—an exploratory study
Aim: This study aims to evaluate the accuracy and readability of responses generated by two large language models (LLMs) (ChatGPT-4 and Gemini) to frequently asked questions by lay persons (the g [...] Read more.

Aim:

This study aims to evaluate the accuracy and readability of responses generated by two large language models (LLMs) (ChatGPT-4 and Gemini) to frequently asked questions by lay persons (the general public) about signs and symptoms, risk factors, screening, diagnosis, treatment, prevention, and survival in relation to oral cancer.

Methods:

The accuracy of each response given in the two LLMs was rated by four oral cancer experts, blinded to the source of the responses. The accuracy was rated as 1: complete, 2: correct but insufficient, 3: includes correct and incorrect/outdated information, and 4: completely incorrect. Frequency, mean scores for each question, and overall were calculated. Readability was analyzed using the Flesch Reading Ease and the Flesch-Kincaid Grade Level (FKGL) tests.

Results:

The mean accuracy scores for ChatGPT-4 responses ranged from 1.00 to 2.00, with an overall mean score of 1.50 (SD 0.36), indicating that responses were usually correct but sometimes insufficient. Gemini responses had mean scores ranging from 1.00 to 1.75, with an overall mean score of 1.20 (SD 0.27), suggesting more complete responses. The Mann-Whitney U test revealed a statistically significant difference between the models’ scores (p = 0.02), with Gemini outperforming ChatGPT-4 in terms of completeness and accuracy. ChatGPT generally produces content at a lower grade level (average FKGL: 10.3) compared to Gemini (average FKGL: 12.3) (p = 0.004).

Conclusions:

Gemini provides more complete and accurate responses to questions about oral cancer that lay people may seek answers to compared to ChatGPT-4, although its responses were less readable. Further improvements in model training and evaluation consistency are needed to enhance the reliability and utility of LLMs in healthcare settings.

Márcio Diniz-Freitas ... Pedro Diz-Dios

Published: November 19, 2024 Explor Digit Health Technol. 2024;2:334–345
DOI: https://doi.org/10.37349/edht.2024.00032
This article belongs to the special issue Digital Health Technologies for the Early Detection of Oral Cancer

Full Text

PDF

View:126

Download:9

Times Cited: 0

Aim:

This study aims to evaluate the accuracy and readability of responses generated by two large language models (LLMs) (ChatGPT-4 and Gemini) to frequently asked questions by lay persons (the general public) about signs and symptoms, risk factors, screening, diagnosis, treatment, prevention, and survival in relation to oral cancer.

Methods:

The accuracy of each response given in the two LLMs was rated by four oral cancer experts, blinded to the source of the responses. The accuracy was rated as 1: complete, 2: correct but insufficient, 3: includes correct and incorrect/outdated information, and 4: completely incorrect. Frequency, mean scores for each question, and overall were calculated. Readability was analyzed using the Flesch Reading Ease and the Flesch-Kincaid Grade Level (FKGL) tests.

Results:

The mean accuracy scores for ChatGPT-4 responses ranged from 1.00 to 2.00, with an overall mean score of 1.50 (SD 0.36), indicating that responses were usually correct but sometimes insufficient. Gemini responses had mean scores ranging from 1.00 to 1.75, with an overall mean score of 1.20 (SD 0.27), suggesting more complete responses. The Mann-Whitney U test revealed a statistically significant difference between the models’ scores (p = 0.02), with Gemini outperforming ChatGPT-4 in terms of completeness and accuracy. ChatGPT generally produces content at a lower grade level (average FKGL: 10.3) compared to Gemini (average FKGL: 12.3) (p = 0.004).

Conclusions:

Gemini provides more complete and accurate responses to questions about oral cancer that lay people may seek answers to compared to ChatGPT-4, although its responses were less readable. Further improvements in model training and evaluation consistency are needed to enhance the reliability and utility of LLMs in healthcare settings.

Open Access

Systematic Review

The use of social media in plastic surgery biomedical research: scoping systematic review
Background: Social media has become ubiquitous; its uses reach beyond connecting individuals or organizations. Many biomedical researchers have found social media to be a useful tool in recruitin [...] Read more.

Background:

Social media has become ubiquitous; its uses reach beyond connecting individuals or organizations. Many biomedical researchers have found social media to be a useful tool in recruiting patients for clinical studies, crowdsourcing for cross-sectional studies, and even as a method of intervention. Social media usefulness in biomedical research has largely been in population health and non-surgical specialties, however, its usefulness in surgical specialties should not be overlooked. Specifically in plastic surgery, social media use to understand patient perceptions, identify populations, and provide care has become an important part of clinical practice.

Methods:

A scoping review was performed utilizing PubMed and Medline databases, and articles were screened for the use of social media as a method of recruitment to a clinical trial, as crowdsourcing (i.e., recruitment for a cross-sectional or survey-based study), or as a method of intervention.

Results:

A total of 28 studies were included, which focused on majority females between 18–34 years old. Despite the ability of the internet and social media to connect people worldwide, nearly all the studies focused on the researchers’ home countries. The studies largely focused on social media’s effect on self-esteem and acceptance of cosmetic surgery, but other notable trends were analyses of patient perceptions of a disease, or surgical outcomes as reported in social media posts.

Discussion:

Overall, social media can be a useful tool for plastic surgeons looking to recruit patients for a survey-based study or crowdsourcing of information.

Amanda Beneat ... Alisha Oropallo

Published: November 15, 2024 Explor Digit Health Technol. 2024;2:313–333
DOI: https://doi.org/10.37349/edht.2024.00031
This article belongs to the special issue Social Media Applications in Biomedical Research

Full Text

PDF

View:177

Download:11

Times Cited: 0

Background:

Social media has become ubiquitous; its uses reach beyond connecting individuals or organizations. Many biomedical researchers have found social media to be a useful tool in recruiting patients for clinical studies, crowdsourcing for cross-sectional studies, and even as a method of intervention. Social media usefulness in biomedical research has largely been in population health and non-surgical specialties, however, its usefulness in surgical specialties should not be overlooked. Specifically in plastic surgery, social media use to understand patient perceptions, identify populations, and provide care has become an important part of clinical practice.

Methods:

A scoping review was performed utilizing PubMed and Medline databases, and articles were screened for the use of social media as a method of recruitment to a clinical trial, as crowdsourcing (i.e., recruitment for a cross-sectional or survey-based study), or as a method of intervention.

Results:

A total of 28 studies were included, which focused on majority females between 18–34 years old. Despite the ability of the internet and social media to connect people worldwide, nearly all the studies focused on the researchers’ home countries. The studies largely focused on social media’s effect on self-esteem and acceptance of cosmetic surgery, but other notable trends were analyses of patient perceptions of a disease, or surgical outcomes as reported in social media posts.

Discussion:

Overall, social media can be a useful tool for plastic surgeons looking to recruit patients for a survey-based study or crowdsourcing of information.

Open Access

Original Article

AI bias in lung cancer radiotherapy
Aim: In lung cancer research, AI has been trained to read chest radiographs, which has led to improved health outcomes. However, the use of AI in healthcare settings is not without its own set of [...] Read more.

Aim:

In lung cancer research, AI has been trained to read chest radiographs, which has led to improved health outcomes. However, the use of AI in healthcare settings is not without its own set of drawbacks, with bias being primary among them. This study seeks to investigate AI bias in diagnosing and treating lung cancer patients. The research objectives of this study are threefold: 1) To determine which features of patient datasets are most susceptible to AI bias; 2) to then measure the extent of such bias; and 3) from the findings generated, offer recommendations for overcoming the pitfalls of AI in lung cancer therapy for the delivery of more accurate and equitable healthcare.

Methods:

We created a synthetic database consisting of 50 lung cancer patients using a large language model (LLM). We then used a logistic regression model to detect bias in AI-informed treatment plans.

Results:

The empirical results from our synthetic patient data illustrate AI bias along the lines of (1) patient demographics (specifically, age) and (2) disease classification/histology. As it concerns patient age, the model exhibited an accuracy rate of 82.7% for patients < 60 years compared to 85.7% for patients ≥ 60 years. Regarding disease type, the model was less adept in identifying treatment categories for adenocarcinoma (accuracy rate: 83.7%) than it was in predicting treatment categories for squamous cell carcinoma (accuracy rate: 92.3%).

Conclusions:

We address the implications of such results in terms of how they may exacerbate existing health disparities for certain patient populations. We conclude by outlining several strategies for addressing AI bias, including generating a more robust training dataset, developing software tools to detect bias, making the model’s code open access and soliciting user feedback, inviting oversight from an ethics review board, and augmenting patient datasets by synthesizing the underrepresented data.

Kai Ding ... Yian Qi

Published: November 12, 2024 Explor Digit Health Technol. 2024;2:302–312
DOI: https://doi.org/10.37349/edht.2024.00030
This article belongs to the special issue Cancer Diagnosis in the Digital Age

Full Text

PDF

View:202

Download:17

Times Cited: 0

Aim:

In lung cancer research, AI has been trained to read chest radiographs, which has led to improved health outcomes. However, the use of AI in healthcare settings is not without its own set of drawbacks, with bias being primary among them. This study seeks to investigate AI bias in diagnosing and treating lung cancer patients. The research objectives of this study are threefold: 1) To determine which features of patient datasets are most susceptible to AI bias; 2) to then measure the extent of such bias; and 3) from the findings generated, offer recommendations for overcoming the pitfalls of AI in lung cancer therapy for the delivery of more accurate and equitable healthcare.

Methods:

We created a synthetic database consisting of 50 lung cancer patients using a large language model (LLM). We then used a logistic regression model to detect bias in AI-informed treatment plans.

Results:

The empirical results from our synthetic patient data illustrate AI bias along the lines of (1) patient demographics (specifically, age) and (2) disease classification/histology. As it concerns patient age, the model exhibited an accuracy rate of 82.7% for patients < 60 years compared to 85.7% for patients ≥ 60 years. Regarding disease type, the model was less adept in identifying treatment categories for adenocarcinoma (accuracy rate: 83.7%) than it was in predicting treatment categories for squamous cell carcinoma (accuracy rate: 92.3%).

Conclusions:

We address the implications of such results in terms of how they may exacerbate existing health disparities for certain patient populations. We conclude by outlining several strategies for addressing AI bias, including generating a more robust training dataset, developing software tools to detect bias, making the model’s code open access and soliciting user feedback, inviting oversight from an ethics review board, and augmenting patient datasets by synthesizing the underrepresented data.
- Submit a Manuscript
- Aims and Scope
- Editorial Board
- Author Instructions
- Article Type
  
  Original Article (2)
  
  Systematic Review (1)

Aim:

Methods:

Results:

Conclusions:

Aim:

Methods:

Results:

Conclusions:

Background:

Methods:

Results:

Discussion:

Background:

Methods:

Results:

Discussion:

Aim:

Methods:

Results:

Conclusions:

Aim:

Methods:

Results:

Conclusions: