8 lessons learned from assessing the performance of apps for mental health

Assessing the adoption, spread and evaluation of apps for mental health

As a clinical psychologist I have been trained to see the value in spoken language and body language at providing insight into a range of mental health concerns. But as digital capabilities evolve, I can also see the value in broadening therapy tools to include apps to deliver information and support in different and innovative ways.

Mental health is one of our priority areas at Health Innovation East and recently we supported a suite of apps which aimed to improve particular aspects of mental health and wellbeing to meet the needs of our region. For example, “how can digital technologies be harnessed to identify and signpost help for those thinking about suicide?” Our team utilised its knowledge of the digital mental health landscape to identify innovators who were able to provide a solution to each of these local challenges.

Understanding and measuring the impact of apps is a particularly problematic area of real-world evaluation, especially when the technology is used by the general population rather than a specific cohort. Through reviewing our evaluations of these apps we have pulled out eight key learnings:

1. One size does not fit all – Do not assume that because a digital solution works for one cohort that it can be transferred to another cohort and have the same success. We were not able to transfer digital solutions to different population groups without modifying the solution. In one particular case we learnt that whilst digital assessment and online CBT in full worked well in one group of young people it didn’t work across a more complex group of young people.
2. Focus on a niche (specifics) – In our experience, apps appear to work best when they are designed for very specific purposes, with clear and honest treatment expectations. For example, we evaluated a cognitive rehabilitation app (known as brain training) to see whether cognitive function and health outcomes of patients could be improved by demonstrating a reduction in the number of attendances at primary and secondary care and improving self-care. Our evaluation found that these outcomes were not met and there was in fact no reduction in attendances. Perhaps these were not the right and realistic outcome measures, leading onto point three.
3. Be rigorous and realistic in your evaluation methodology – We had two projects that aimed to intervene and stop the escalation of suicidal thoughts. The very nature of an intervention at a general population level in such a complex area with many confounding variables was always going to be difficult to measure. Our evaluation was unable to demonstrate any behavioural change however the digital tools in both cases were well used so we will never know the impact they had on the individuals who used them.

Reviewing this suite of apps has also clarified other challenges including:

4. Navigating governance – the additional difficulties around governance, in particular risk management, with apps sitting outside of formal clinical pathways.
5. Confidence in apps – the sheer number of apps that are available erodes confidence and trust. It is hard and time consuming for professionals to differentiate between safe and unsafe apps, or helpful and unhelpful apps, and to understand the evidence base for each.
6. Systems and processes – although there is an improving picture from the rapid digitisation of health and consultations as a result of COVID-19, systems and processes are not fully enabled to integrate apps and so we are perhaps some way off healthcare professionals embedding digital health apps into day to day working practices.
7. Awareness and habit -apps are not yet part of formal training of mental health professionals and this may inhibit professionals’ confidence to use them as part of standard care.
8. Return on investment – evaluation of impact (both individually and on the wider system) remains a challenge and so inhibits adoption (particularly if immediate or short-term returns are sought).

What does the future hold for apps for mental health?

There is great potential for digital innovations to add value to clinical populations where the human element of care is enhanced, to complement but not replace the therapeutic relationship. However, it is important to not see digital apps as a short-term panacea for long-standing mental health problems.

At a time of social distancing and restricted social activity, it is important to recognise the value of digital innovation and its ability to deliver therapeutic interventions remotely and also to support the general population in its wellbeing and self-care. Furthermore, the accessibility of apps may provide opportunities for those who ordinarily would be labelled in mental health services as “reluctant to engage”, as they may be more willing to engage with digital services accessible in their home environment.

It is clear that digital innovations for mental health have a critical role to play in delivering support to those who need it, as highlighted in the Long-Term Plan for the NHS. At a national level we need innovative ideas that can be supported, adopted and evaluated. A national apps library and NHS approved apps provide a central point for accessing trusted apps, and following the COVID-19 pandemic ORCHA (Organisation for the Review of Care and Health Applications) advises which apps are safe and effective. This will be helpful for healthcare professionals to decide which apps to support and recommend to others. It is however key that robust evaluations of digital innovations continue, to ensure we can reflect and learn from what is working well (and not working so well), and it will certainly be interesting to see how digital innovations evolve in the coming years.