8 lessons learned from assessing the performance of apps for mental health

Dr Sarah Robinson, Director of Delivery, reflects on her experience evaluating apps for mental health

Published: 10th October 2020

Assessing the adoption, spread and evaluation of apps for mental health

As a clinical psychologist I have been trained to see the value in spoken language and body language at providing insight into a range of mental health concerns. But as digital capabilities evolve, I can also see the value in broadening therapy tools to include apps to deliver information and support in different and innovative ways.

Mental health is one of our priority areas at Health Innovation East and recently we supported a suite of apps which aimed to improve particular aspects of mental health and wellbeing to meet the needs of our region. For example, “how can digital technologies be harnessed to identify and signpost help for those thinking about suicide?” Our team utilised its knowledge of the digital mental health landscape to identify innovators who were able to provide a solution to each of these local challenges.

Understanding and measuring the impact of apps is a particularly problematic area of real-world evaluation, especially when the technology is used by the general population rather than a specific cohort. Through reviewing our evaluations of these apps we have pulled out eight key learnings:

  1. One size does not fit all – Do not assume that because a digital solution works for one cohort that it can be transferred to another cohort and have the same success. We were not able to transfer digital solutions to different population groups without modifying the solution. In one particular case we learnt that whilst digital assessment and online CBT in full worked well in one group of young people it didn’t work across a more complex group of young people.
  2. Focus on a niche (specifics) – In our experience, apps appear to work best when they are designed for very specific purposes, with clear and honest treatment expectations. For example, we evaluated a cognitive rehabilitation app (known as brain training) to see whether cognitive function and health outcomes of patients could be improved by demonstrating a reduction in the number of attendances at primary and secondary care and improving self-care. Our evaluation found that these outcomes were not met and there was in fact no reduction in attendances. Perhaps these were not the right and realistic outcome measures, leading onto point three.
  3. Be rigorous and realistic in your evaluation methodology – We had two projects that aimed to intervene and stop the escalation of suicidal thoughts. The very nature of an intervention at a general population level in such a complex area with many confounding variables was always going to be difficult to measure. Our evaluation was unable to demonstrate any behavioural change however the digital tools in both cases were well used so we will never know the impact they had on the individuals who used them.

Reviewing this suite of apps has also clarified other challenges including:

  1. Navigating governance – the additional difficulties around governance, in particular risk management, with apps sitting outside of formal clinical pathways.
  2. Confidence in apps – the sheer number of apps that are available erodes confidence and trust. It is hard and time consuming for professionals to differentiate between safe and unsafe apps, or helpful and unhelpful apps, and to understand the evidence base for each.
  3. Systems and processes – although there is an improving picture from the rapid digitisation of health and consultations as a result of COVID-19, systems and processes are not fully enabled to integrate apps and so we are perhaps some way off healthcare professionals embedding digital health apps into day to day working practices.
  4. Awareness and habit -apps are not yet part of formal training of mental health professionals and this may inhibit professionals’ confidence to use them as part of standard care.
  5. Return on investment – evaluation of impact (both individually and on the wider system) remains a challenge and so inhibits adoption (particularly if immediate or short-term returns are sought).

What does the future hold for apps for mental health?

There is great potential for digital innovations to add value to clinical populations where the human element of care is enhanced, to complement but not replace the therapeutic relationship. However, it is important to not see digital apps as a short-term panacea for long-standing mental health problems.

At a time of social distancing and restricted social activity, it is important to recognise the value of digital innovation and its ability to deliver therapeutic interventions remotely and also to support the general population in its wellbeing and self-care. Furthermore, the accessibility of apps may provide opportunities for those who ordinarily would be labelled in mental health services as “reluctant to engage”, as they may be more willing to engage with digital services accessible in their home environment.

It is clear that digital innovations for mental health have a critical role to play in delivering support to those who need it, as highlighted in the Long-Term Plan for the NHS. At a national level we need innovative ideas that can be supported, adopted and evaluated. A national apps library and NHS approved apps provide a central point for accessing trusted apps, and following the COVID-19 pandemic ORCHA (Organisation for the Review of Care and Health Applications) advises which apps are safe and effective. This will be helpful for healthcare professionals to decide which apps to support and recommend to others. It is however key that robust evaluations of digital innovations continue, to ensure we can reflect and learn from what is working well (and not working so well), and it will certainly be interesting to see how digital innovations evolve in the coming years.

National NHS wellbeing support

The NHS nationally and locally has developed a range of wellbeing support to care for and protect all NHS people, whether at the frontline or in supporting services.

All NHS staff now have access to a range of support (#OurNHSPeople Wellbeing Support) through one point of contact:

– a free wellbeing support helpline 0300 131 7000, available from 7.00 am – 11.00 pm seven days a week, providing confidential listening from trained professionals and specialist advice – including coaching, bereavement care, mental health and financial help

– a 24/7 text alternative to the above helpline – simply text FRONTLINE to 85258

– online peer-to-peer, team and personal resilience support, including through Silver Cloud, and free mindfulness apps including Unmind, Headspace Sleepio and Daylight


We are encouraging NHS teams to take immediate advantage of these services. They can be used in addition to the support available from your own organisation and as stated in the bulletin below, will develop further in the coming weeks in line with user feedback (nhsi.wellbeingc19@nhs.net).


Our people -Sarah Robinson Head of Delivery Health Innovation East
About me

Sarah has held numerous strategic NHS roles since qualifying as a clinical psychologist. She was the regional lead for the national dementia strategy before taking up a patient safety and safeguarding leadership role at NHS England.

Sarah oversees the local delivery of Health Innovation East’s national programmes and national products, on which we spend around 50% of our time. This includes supporting organisations in the region to embed proven innovations such as those selected by the AHSN Network for adoption, including ITP and Rapid Uptake Products and those from the NHS Accelerated Access Collaborative. Her portfolio also includes oversight of local projects borne out of responding to local challenges that the NHS has identified that Health Innovation East have matched and supported through innovative practice.

You may also like…

Tara Marshall Health Innovation East Clinical Lead for Deterioration
Four things I learned working on the intensive care ward during COVID-19

Tara Marshall, Clinical Lead for Deterioration, reflects on her return to ICU during coronavirus. Read more.

Dr Jag Alhuwalia Health Innovation East Chief Clinical Officer
Managing the risks of success through clinical governance

Dr Jag Ahluwalia, Chief Clinical Officer at Health Innovation East, explains how our approach to clinical governance minimises risk in innovation. Read more.

Michael Hornberger Health Innovation East Board talks about citizen science
Tapping into citizen science

Prof Michael Hornberger discusses strengthening academic research through partnerships and public involvement. Read more.

Tell us your healthcare challenges

We know you want access to the latest innovations to improve patient care and enable efficiencies.

Get involved