Abstract
Aim:
Mobile health applications (MHAs) have been rapidly designed and urgently need evaluation. Existing evaluation methods, such as platform, development, and subjective overall user observations, are mostly based on application (app) design. This study aimed to evaluate the functionality and feasibility of an MHA to train cognitive function in youth at clinical high risk (CHR) for psychosis with a tool that allows a comprehensive user experience evaluation of mobile apps from multiple dimensions.
Methods:
Eighty participants with CHR for psychosis were recruited and randomly assigned to the intervention and the group control. Participants in the intervention group used the Specific Memory Attention Resource and Training (SMART) app for three months. MHA’s functionality and feasibility were measured by the mobile app rating scale (MARS) and qualitative tools.
Results:
Participants in the SMART group report that the form and design of this app are simple to operate, and the content is trustworthy. They reported improvement in cognitive function and more motivation to seek help to improve their cognitive function. They also pointed out areas of improvement.
Conclusions:
SMART usability and functionality were measured by a multidimensional tool. It shows promise in improving CHR memory and attention and demonstrates appropriate usability and functionality.
Keywords
User interface design, mobile health application, serious games, mental health, user willingnessIntroduction
Human factors engineering involves understanding the need for comprehensive integration with the popularization and development of smartphones so patients can break geographical and socio-economical barriers and get higher-quality health services [1, 2]. At present, mobile healthcare has entered a vigorous development stage. The Groupe Speciale Mobile Association estimated that the United States and China were the largest countries in the mobile medical market in 2022 [3]. Most of the mobile health (mHealth) services are for chronic illness management, followed by diagnosis and treatment [4–6]. The large amount of capital entering the mHealth market shows that mHealth will be a market with great potential, and it also means that this health service has practical and professional implications. There will be 4.3 billion smartphone users in 2023 in the world [7]. The smartphone application (app) market has enormous growth potential. By using smartphones, health service providers and users can easily connect, saving time and travel costs and allowing doctors to treat patients more effectively and efficiently [8].
In the field of mental health, mental illnesses tend to be chronic and highly stigmatizing, especially in Asian cultures [9]. Convenient, non-stigmatizing, and engaging mental mHealth intervention approaches can effectively reduce the burden of disease and improve the compliance of patients [10, 11]. However, there is a lack of symptom-specific mHealth apps to improve patient functioning. For example, in patients with schizophrenia, substantial deficits in everyday functioning are common and related to cognitive deficits such as compromised memory and attention [12]. Furthermore, cognitive deficits (e.g., memory and attention) are resistant to change with psychiatric medications, which mainly target positive symptoms (delusional thinking, hallucination, etc.) and negative symptoms (flat affect, lack of motivation, etc.) of the illness. Early intervention of individuals at clinical high risk (CHR) for psychosis or schizophrenia that improves patients’ cognitive function would be critical in improving treatment and functional outcome and, potentially, preventing transition to psychosis. Therefore, the obvious need and urgency for prevention and intervention of mental illnesses like schizophrenia and the need to conduct quality research projects have created an unprecedented opportunity for health services and care involving mHealth apps (MHAs).
mHealth care has inherent advantages, such as easy access and reduced mental health prevention and intervention stigma. Surveys of mental health patients found that most patients support accessing health services and self-management through a smartphone [5, 6]. With the help of a cognitive training program via smartphone, schizophrenia patients showed improved self-efficacy, mood, and motivation [13]. One MHA for CHR also indicated that it is feasible for smartphone programs to assist in monitoring the symptoms of CHR [14]. Designing MHA for CHR can help to test the efficacy of MHA-based cognitive training programs to improve CHR cognitive functions, thereby helping CHR to obtain health services more conveniently, quickly, and in real-time [12]. Furthermore, compared with the schizophrenia stage, cognitive function intervention at the CHR stage can achieve a better prognosis [8].
Furthermore, the smartphone cognitive training program can help patients and their families reduce financial burdens, break space, and distance restrictions, and help mental health professionals and patients communicate more conveniently, saving enormous travel and wait time [15]. Thus, the development of smartphone apps is not only a general trend but can also meet patients’ actual needs [16]. The development of mHealth programs that can improve cognitive function not only enriches the way of cognitive training but also helps determine adequate training time and training forms and recognize their potential therapeutic effects [12, 17].
At present, apps used to monitor and manage health have been using recommendation algorithms on the Apple Store [18]. The main recommendation method on the app recommendation page is “popular app/using by everyone/fresh app”, but this recommendation method is based on the users’ overall rating and evaluation; it lacks objectivity [19]. At present, a large number of health apps have been designed and are in urgent need of evaluation. Still, the past evaluation methods are mostly based on app design, such as platform, development, and subjective overall user observations [20]. There is a great need to obtain users’ feedback on specific features of apps using more standard and well-validated evaluation instruments [19]. In a study conducted by Kelly et al. [21] to examine the feasibility and acceptability of a Specific Memory Attention Resource and Training (SMART) track app, researchers obtained user feedback through qualitative phone interviews. However, only 28% of initial participants provided feedback, which may not fully represent a valid and reliable app evaluation [21]. Other studies tend to measure the app’s limited dimensions (content and acceptability) [22]. Furthermore, many measures used to evaluate feasibility, acceptability, and functionality tend to be qualitative and lack psychometric information [22]. Therefore, this study aimed to use a widely used instrument to evaluate the functionality and feasibility of an MHA to train cognitive function in CHR. This instrument allows a comprehensive evaluation of user experiences of mobile apps from multiple perspectives [23]. A well-developed and validated mHealth app allows improved performance in human workload, ease of maintenance, and personnel safety [24].
Materials and methods
Participants and procedures
Eighty CHR patients were recruited from 2017 to 2019 continuously in two hospitals in China: Shanghai Mental Health Center (SMHC, n = 65) and Suzhou Guangji Hospital (n = 15). Similar recruitment and assessment procedures were applied to obtain participants across the two sites. Forty participants were randomly assigned to the SMART 1.0 app group and 40 participants to the control group as usual in their naturalistic environment. Please refer to Li et al. [12] (2021) for detailed information about participant inclusion criteria and demographic information. The research ethics committee of SMHC approved the study, and participants gave written consent or permission before participating in the research process.
Instruments
SMART app
Participants did cognitive training (memory and attention in particular) in an MHA called SMART, an app modified based on Learn, Assess, Manage, and Prevent (LAMP) [6, 25].
SMART is available in both English and Chinese and iPhone operating system (IOS) and Android versions. Participants in this study used the IOS version. SMART consists of different components, including nine memory games (n-back, spatial span, simple memory, series 7, visual-spatial tasks, digit span forward and backward, cats and dogs, time series, and n-back new) and five attention games [trails B new, trails B, jewelry A, jewelry B, and three-dimensional (3D) image]. In order to engage participants and reduce attrition, SMART used an app programming interface (API) allowing participants to get monetary rewards via redpockets, a fun and traditional Chinese way of receiving money as a gift, a feature reflecting cultural sensitivity. Please see Li et al. [12] (2021) for detailed descriptions of SMART components.
Mobile app rating scale
Mobile app rating scale (MARS) [21] consists of three components: app quality evaluation, app subjective evaluation, and perceived impact. MARS rates app quality with 19 items in four dimensions: app participation, function, aesthetics, and information. All items are rated on a five-point Likert scale, from “1-inadequate” to “5-excellent”. Each section is scored with the mean (M) score. The total score of the app quality evaluation component is the average score of the total scores of the four dimensions:
Engagement: Evaluate whether the game is fun, attractive, customizable, interactive, with little reminders (e.g., sending alerts, messages, notifications, feedback, and enabling sharing), and meeting the needs of users. For example, item 3, “Customize: Does it allow you to customize your favorite settings and preferences (such as sound, content, notifications, etc.)?”.
Functionality: Evaluate app function, navigation, flow of logic, gestural design of the app, and whether it is easy to learn. For example, item 8, “Navigation: Is the switch between pages logical/accurate/appropriate/unobtrusive? Are all necessary links established between screens?”.
Aesthetic: Evaluate the graphic design, overall visual attraction, color scheme, and consistency of style of the app. For example, item 10, “Layout: Are the layout and size of buttons/icons/menus/content on the screen appropriate? Can they be zoomed if necessary?”.
Information: Evaluate whether the app contains high-quality information from reliable sources (such as text, feedback, measures, and references). For example, item 16, “The credibility of the source: Does the information in the app come from a trusted source (description in the app store or in the app)?”
The app’s subjective quality dimension contains four questions, asking the users if they are willing to recommend this MHA to others, how many times they will use this MHA in the next year, whether they are willing to pay for it, and what their star rating is. The five-point scoring system is used to evaluate subjective quality. A higher number indicates more willingness to use the app.
The section on the perceived impact includes the impact on the evaluator’s consciousness, knowledge, attitude, motivation, seeking help, and behavior change of using this app. A five-point scoring system, from strong disagreement to strong agreement, is also used.
SMART improvement survey
Besides MARS, a set of questions was created to seek participants’ feedback on how to improve SMART and other similar cognitive training MHAs. A sample question is: If you could change one thing about this app, what would it be? These survey results will help us to improve SMART from interface design to game selections and designs.
Statistical analyses
Statistical Package for the Social Sciences 26 was used to conduct data analyses [26].
Results
Participant information
This present study analyzed results from the intervention group that used SMART. The age range of the 40 participants in the intervention group was 14 to 38 [M = 20.51, standard deviation (SD) = 5.89], with education level ranging from primary school to university, with average years of education at 11.54 years (SD = 2.80). Thirty-six participants were right-handed. Ten out of the 40 participants were lost in the three-month follow-up, leaving 30 participants (15 males, 15 females) for the current analyses. Reasons for attrition include inconvenient time or remote location, refusal to be evaluated at the follow-up without a specific explanation, and change of contact information. No significant differences were found between the CHRs at recruitment and those who successfully finished the follow-up (n = 30).
MARS results
MARS has three components. Detailed results of the first component-app quality evaluation have been reported elsewhere [12]. The overall result indicates that participants rated the information dimension the highest, followed by function, aesthetics, and engagement. Participants felt that the information in SMART was trustworthy and well-illustrated using charts, graphs, and images. In addition, the SMART components primarily work well, are easy to learn and understand, and have clear instructions. Switching between screens is accurate and appropriate, but shortcuts are lacking. Gestures such as tapping, swiping, and turning pages are the same on each page. Participants felt it was neither fun nor boring for the engagement dimension and could engage users for about 5 min. They wish there were more interactive features between users and SMART.
There are some key findings about the responses to the four questions about subjective evaluation for SMART. Most CHR participants (96.55%) reported they might recommend SMART to others (Figure 1). More than half of the participants (51.75%) indicated that they will use the SMART more than ten times in the next year (Figure 2). At the same time, a considerable portion of CHR participants (44.84%) would not pay to use SMART (Figure 3).
The results also show that participants enjoyed or did not favorite SMART—in terms of things they liked about SMART, participants’ favorite points of the SMART app mainly focused on simplicity, convenience (24%), and improved memory, attention, and responsiveness (21%) in Figure 4. For the shortcomings of SMART, 24% of users think that SMART is too simple, not attractive enough, or it takes too long and time-consuming to use. Another 17% of users think this app’s title is fixed and repetitive (Figure 5). About 10% of users think this app’s user interface table and the game content are rich and diversified (Figure 4).
Regarding who would like this app, 41% of users think that children and teenagers will like this app, 28% of users believe that people who need to improve their attention will like this app, and 10% believe that the elderly will like this app (Figure 6). Other 21% of users indicate that people with cognitive impairment, poor memory, and dissatisfaction with their brain abilities would like this app (Figure 6). For those who benefit from this app, 41% of users believe that people with low awareness and inattention will benefit. About 24% of users believe it will help children and young people. Four percent think that older people will benefit from it. The answers from the other 31% of users consider people who are sick, and developers will benefit from it (Figure 7). As for the circumstances under which users are willing to use the app, 34% reported other reasons like wanting to feel better, 28% of users said they would like to use it when they are bored, 24% of users said they will use it when they have free time, and 14% of users said they will use the app when their memory is not good (Figure 8). In terms of why users want to encourage friends to use this app, 48% of users will recommend this app because it can help users improve their memory. About 7% of users recommend it as it can help users’ health. The answers of the other 45% of users include sticking to one thing, fascinating software, and so on (Figure 9).
Regarding the problems encountered when participants were using the app, 6 users out of 40 indicated that there was a slow response, and the program stuck, while four users said that the flashback data could not be saved. Other problems include not recording the results of the use when the network was not good, multiple bugs, not responding, no new ideas, and slow submissions (Figure 10).
To summarize the problems users encountered in Figure 10, there are two categories of problems: (i) network speed for the SMART data storage, (ii) there are not enough SMART games for users to select.
To enhance the SMART mHealth app and address the identified issues, several key improvements and strategies can be implemented: (i) address the 1st category problem of slow response and program freezing by optimizing the app’s code and improving its efficiency. Enhance the overall speed and responsiveness of the app to provide a seamless user experience. In addition, by implementing a robust data storage and retrieval system, (ii) address the 2nd category problem of encouraging and gathering ideas from users for new features or improvements that can enhance the app’s functionality and user engagement. Finally, a user feedback mechanism within the app can be implemented to allow users to report issues and suggest improvements.
Use user feedback as a valuable resource for identifying and resolving problems promptly.
According to MARS results, after three months of SMART training, as shown in Table 1, in terms of consciousness, knowledge, attitude, changing motivation, seeking help, and behavior change, the participants in the SMART group’s CHR reported that SMART can improve cognitive function.
Patient improvement with SMART in three months
| Improvement | Mode | Median | M | SD |
|---|---|---|---|---|
| Consciousness | 3 | 3 | 3.28 | 1.25 |
| Knowledge | 3 | 3 | 3.24 | 1.19 |
| Attitude | 3 | 3 | 3.24 | 1.19 |
| Changing motivation | 3 | 3 | 3.34 | 1.23 |
| Seeking help | 4 | 4 | 3.45 | 1.15 |
| Behavior changes | 3 | 3 | 3.38 | 1.15 |
Overall evaluation of SMART
With regard to the subjective evaluation of the quality of SMART, CHR participants believe that the quality of the app is acceptable and meets the general functional requirements (Table 2). However, there are deficiencies in the designing and planning of user groups. In terms of using the app, the respondents recommend this app to others and may continue to use the app in the future but will not pay for it. After three months of training, CHR’s motivation to seek help for their cognitive symptoms has increased, and they can adopt a more positive attitude when they are facing cognitive problems.
Overall evaluation of SMART
| Access to amenities | Minimum | Maximum | M | SD |
|---|---|---|---|---|
| App quality evaluation | 2.78 | 5.00 | 3.72 | 0.46 |
| App subjective evaluation | 1.00 | 4.50 | 2.91 | 0.85 |
| Perceived impact | 1.00 | 5.00 | 3.32 | 1.01 |
Summary
From the function of SMART app, CHR thinks that the content of SMART is scientific, the form and design of this app are simple, and the function of cognitive training is distinct. However, the interface of the SMART app can be more attractive, and it is entertaining. The design of the SMART app needs to learn more from the currently popular game apps to make SMART more competitive and interesting and attract users to spend more time on those training games voluntarily. For usability, CHR participants provided more positive evaluations of the SMART app, they would be willing to continue to use the app in the future, and ready to recommend it to others. But few people are willing to pay for this app. In the future, more functions should be added, such as monitoring health status, tracking medication use, sending health information, and the WeChat binding push. The SMART app can continue to use the daily message reminders and monetary rewards currently used, but the form can have more integration with the app, establishing a user community, implementing the form of rewards, and increasing interactivity between users. In addition, more individualized features, such as selected education videos based on each user’s progress and specific progress stages, can be added to this app.
Discussion
Current mHealth apps lack targeted symptom-specific and mental health-relevant features and characteristics [27]. In the absence of an adequate level of research, there are several places that clinicians can turn to for assistance in assessing the suitability of specific mHealth apps [23]. So, there is an urgent need for medical and mental health professionals to work closely with software developers and develop age-appropriate and symptom-focused mHealth training apps. Those apps will be more effective and useful for worldwide users and patients. In addition, mental health clinical and research programs can integrate a mHealth program via Bluetooth to fitness trackers as well as other devices such as wearables. Such devices can also contain novel sensors, which could improve users’ health observations, such as medication, blood pressure, sweating, and sleep patterns. This will allow researchers to investigate the association of physiological symptoms with mental health challenges [28–31].
The problem of ensuring mHealth app users’ security and privacy is also of utmost importance and requires significant alignment among legislation, industry practices, and consumers (patients or health enthusiasts) [32]. Current solutions include policy-based access to controlled data. The problem could be better addressed through adequate coordination between global and governmental agencies as well as industry consortia. In particular, the app developer community could benefit from a more streamlined process of recording and linking mHealth app vulnerabilities to specific regulations and standards. mHealth apps are different from other mobile apps, as they are subject to more global, governmental, and industry standardization [33]. Oversight and regulations are needed mainly for patient well-being and user security and privacy reasons. Standardization is driven by multiple factors, including efficient sharing of medical data between providers, patients, and third parties, as well as platform compatibility between mHealth apps and cloud services. The limitations and constraints need to be addressed through the recording process, ensuring that vulnerability recordings not only describe the nature of the security and privacy vulnerability in question but also its particular relevance to specific governmental regulations, industry standards, app developers, and proprietary device implementations [34]. Of course, security and privacy are always priorities for mHealth apps since personal health information is part of mHealth app data. Most of the mHealth apps, however, have ignored this issue. This issue on privacy and security with mHealth apps warrants imminent research and exploration.
In conclusion, mHealth apps have the potential to provide more convenient healthcare services, improve or maintain the health status and quality of life of patients and their family members, and promote the development of the health industry as well as the health environment. Encrypted mobile devices are the best and easiest way to protect users’ privacy. Therefore, designing and developing secure, more cognitive function training apps in inexpensive mobile devices that are culturally sensitive is in huge demand with the fast growth of mHealth fields.
Abbreviations
| app: | application |
| CHR: | clinical high risk |
| M: | mean |
| MARS: | mobile application rating scale |
| MHAs: | mobile health applications |
| mHealth: | mobile health |
| SD: | standard deviation |
| SMART: | Specific Memory Attention Resource and Training |
Declarations
Acknowledgments
We thank Ms. Breanna Jones for reviewing and reformatting in-text citations and references.
Author contributions
HL and HC: Conceptualization, Investigation, Writing—original draft, Writing—review & editing. SY: Data curation, Formal analysis, Methodology, Writing—review & editing. LX and TZ: Data curation, Formal analysis, Writing—review & editing. FB: Writing—review & editing. WSS: Data curation, Writing—review & editing. JW: Project administration, Writing—review & editing. All authors have reviewed, discussed, and agreed to their individual contributions.
Conflicts of interest
The authors declare that they have no conflicts of interest.
Ethical approval
This study was approved by the SMHC ethics committee and complied with the Declaration of Helsinki.
Consent to participate
Informed consent to participate in the study was obtained from all participants.
Consent to publication
Not applicable.
Availability of data and materials
The datasets generated and analyzed for this study can be requested via first and corresponding author.
Funding
Huijun Li is supported by “Enhancing Intervention of Attenuated Psychosis Syndrome with M-Health Technology” from National Institute of Mental Health [R21 1 R21 MH113674-01A1]. The funder supported study design, data collection, analysis and interpretation, and writing of the report. The funder had no role in decision to publish.
Copyright
© The Author(s) 2024.
PDF
