The potential of AI technology for including experience-based knowledge and values in vaccination guidelines

Lea Lösch PhD candidate, Athena Institute

How might AI technology innovate the inclusion of experience-based knowledge in clinical practice guidelines and why is there a need for this? The joint project of VU and RIVM “Evidence in Action” explores this potential using the example of vaccination guidelines. Initial results give an impression of how AI-based methods may be beneficial, but also show why AI is only one part of the solution.

A time before evidence-based medicine and AI

As a patient seeking medical advice in the 1980s, different health professionals may have given you very different suggestions for treatment. This resulted in a large variety of treatments for the same diagnoses which was not always clinically justified, leading to suboptimal care. You might not receive the most effective treatment even though it exists. For example, obstetrician Graham Liggins and paediatrician Ross Howie already published a paper in 1972 which clearly showed that more premature babies survived when their mothers took steroids. However, many physicians did not know about this or did not realise its effectiveness until the Royal College of Obstetricians and Gynaecologists published clinical guidelines based on a systematic review of various studies and trials.

Evidence-based medicine

This situation gave rise to the concept of evidence-based medicine (EBM). EBM aims to make healthcare more standardized and systematized based on up-to-date research – ensuring that you receive the best care regardless of where you go to see a doctor. In 1996, Sackett and colleagues defined EBM as ‘‘the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients” [1]. As shown in the figure below, they envisaged drawing on evidence from 1) the best available clinical research, 2) individual clinical expertise, and 3) patient values and preferences.

Experiential and expert knowledge in the form of clinical expertise and patient values have thus always been an integral part of EBM. In practice, however, clinical research, mainly from systematic reviews and randomised control trials (RCTs), has been established over time as the predominant source of knowledge – as the “gold standard”.

Integrating experiential knowledge in clinical guidelines

Published clinical research is also the primary source of evidence for developing clinical practice guidelines. Clinical practice guidelines – such as the ones mentioned above for the use of steroids to reduce neonatal morbidity – have been developed to bring together the best available evidence on a given issue to support the practitioner in clinical decision-making on the best treatment option.

Research over the past two decades has shown that the integration of more diverse knowledge next to clinical research into guideline development has the potential to lead to more suitable, helpful and robust guidelines [2]. Patients and health professionals can provide unique perspectives and recommendations based on their experiences regarding for example feasibility, relevance, prioritisation and preference. Incorporating these insights into guideline development can result in guidelines that better align with what is happening in practice [3].

A persistent challenge

Despite these benefits, the meaningful inclusion of experiential knowledge and value considerations into guideline development remains a challenge. Among the main reasons for this are epistemic and methodological difficulties.

Epistemic tensions

As explained above, EBM and clinical guideline development rely predominantly on meta-reviews of randomized clinical trials. They thus draw primarily on frequentist reasoning to develop a population-based guideline. The first challenge then arises when attempting to articulate a patient’s experience-based knowledge in this epistemic setting [4]. Patients more often contribute qualitative knowledge, which is very vulnerable in this context. It is easily dismissed instead of being listened to, acknowledged and included in the evidence base.

Methodological difficulties

Next to epistemic tensions, a lack of suitable methods and infrastructures is often an obstacle to the integration of experiential knowledge. The tools used by guideline developers to improve consistency and streamline guideline development are very inflexible and, in most cases, not designed for patient participation or integration of diverse knowledge.

Current methods for patient and health professional involvement are for example interviews, surveys, consultation workshops, or reviews of draft versions. Depending on the level of participation, patients need to be knowledgeable about the guideline development process. This leads to the professionalization of patient representatives and at the same time to doubts about the extent to which they are still representative of the ‘average’ patient [4]. The inclusion of experiential knowledge becomes even more difficult in the case of vaccination guidelines. The target population for vaccination is usually the whole population and no selected group of patients who could be invited to the table for guideline development.

It is especially in relation to this second challenge related to methodological issues that AI technology has the potential to contribute to a solution.

AI as a potential game-changer

New computational methods and the availability of large amounts of data open up new opportunities for the inclusion of experience-based knowledge and value considerations in guideline development. For the first time, experiential knowledge and value judgements could be extracted and analysed on a large scale, systematically and potentially in an automated way, not only after, but also alongside the process of guideline development and appraisal of new information. AI-based methods may thus enable a more adaptive approach to incorporate experience and value judgements into guideline development.

The Evidence in Action project by the Athena Institute and Social AI department of the Vrije Universiteit Amsterdam (VU) and the National Coordination Centre for Communicable Disease Control (LCI) of the National Institute for Public Health and the Environment (RIVM) explores how these AI-based methods may innovate the inclusion of experiential knowledge and value considerations in clinical guidelines for vaccination, including the Dutch COVID-19 vaccination guideline. We mobilise AI-methods particularly from the field of natural language processing (NLP) to extract and analyse existing but unexplored sets of textual records where societal and professional knowledge and concerns are expressed, such as databases from the RIVM as well as various social media platforms. Eventually, the aim is to translate the gathered knowledge into robust knowledge that can be integrated into the development of vaccination guidelines in close cooperation with the guideline development group.

How AI can be part of the solution

The project is still ongoing and in line with the dynamics and speed of developments concerning COVID-19, we are still learning new aspects every week. However, initial results can already give an impression of the ways in which AI-based methods may be helpful for integrating citizens’, health professionals’ and patients’ experiential knowledge into (vaccination) guidelines.

First of all, AI-methods can support the researcher or guideline developer in identifying experiential knowledge in texts. Filtering techniques combined with machine learning models can, for example, detect and select comments on social media in which people describe their experiences with COVID-19 vaccination.

In addition, automated text analysis methods such as topic modelling can be used to cluster discussions on Facebook or questions received by the RIVM hotline into a number of topics, providing an overview of the most frequently addressed issues.

Lastly, we also experimented with using NLP methods to support the guideline development group in making informed decisions by allowing a real-time checking of the prevalence of an issue. For example, during a meeting, it was unclear whether the course of action for Dutch citizens who had already been fully vaccinated with a vaccine not approved by the EMA should be included in the vaccination guideline, or whether it was such a minor issue that it could be explained in the FAQ. Through intelligent filtering of the questions received by the RIVM hotline, it was quickly clarified that this was an issue that only affected a few citizens and therefore did not need to be included in the guideline. 

Translating AI findings to guideline development

The above-mentioned examples provide a first impression of how AI methods may contribute to the inclusion of citizens’, patients’ and health professionals’ experience-based knowledge and value considerations in vaccination guidelines. However, providing the infrastructure and making it possible to access and analyse experiential knowledge and value considerations in big data, is only half of the battle.

AI methods may overcome methodological hurdles, however, it is questionable to what extent AI can resolve the persistent epistemic tensions described earlier. What counts as legitimate knowledge is a shared understanding of a distinctive community, in this case primarily guideline development professionals. “Such communities provide a context of justification for what a particular community finds reasonable, true, and useful” [5]. AI can help a community gain access to experiential knowledge; however, this does not automatically ensure a community’s acceptance of this type of knowledge. Exploring the translations between AI renderings of experiential knowledge and guideline development practices may thus be the true transdisciplinary challenge of this research project.

——————————————-

[1] Sackett, D., Rosenberg, W., Gray, J. A. M., Haynes, B., & Richardson, W. S. (1996). Evidence-based medicine: What it is and what it isn’t. British Medical Journal, 312, 71–72.

[2] Armstrong, M. J., Mullins, C. D., Gronseth, G. S. & Gagliardi, A. R. (2018). Impact of patient involvement on clinical practice guideline development: a parallel group study. Implementation Science, 13(1), 55. https://doi.org/10.1186/s13012-018-0745-6

[3] Moleman, Marjolein, Sonja Jerak-Zuiderent, Hester van de Bovenkamp, Roland Bal & Teun Zuiderent-Jerak (in press). Evidence-basing for quality improvement; bringing clinical practice guidelines closer to their promise of improving care practices.  Journal of Evaluation in Clinical Practice.

[4] Bovenkamp, H. M. van de & Zuiderent‐Jerak, T. (2015). An empirical study of patient participation in guideline development: exploring the potential for articulating patient knowledge in evidence‐based epistemic settings. Health Expectations, 18(5), 942–955. https://doi.org/10.1111/hex.12067

[5] Collins, P. H. (2019). Intersectionality as Critical Social Theory. Duke University Press.