Case study - Robot analyst

Professor Sophia Ananiadou

Department of computer Science, University of Manchester

What is the background of your methodology research?

Evidence-based medicine uses systematic reviews to identify relevant studies to answer specific research questions. Such reviews have a central role in health technology assessments, clinical and public health guidelines, and evidence-informed policy and practice. This guidance informs a number of sectors, such as NHS, local authorities and the wider public.

The approach involves: searching to identify all relevant studies for inclusion; screening to filter results further for relevance; and synthesizing of evidence from all relevant included sources. The PICO framework is conventionally used to structure pre-defined research questions matching clinical needs, helping to identify the Population, the Intervention, the Comparator and the Outcome.   However, PICO is not well-suited to the needs of evidence-based public health reviews such as those conducted by NICE. Public health questions are always complex, involving behaviour, culture and organizations, and often are described using abstract, fuzzy terminology in ways that make binary definitions of all the parameters in PICO a priori extremely problematic. Instead, dynamic and multidimensional definition of relevance is required.

We thus developed novel research methods integrated in a system, the RobotAnalyst, which supports users to screen while searching. Its text mining based functionalities, allow users to search via different facets (terms, journals, authors, etc.), via clusters (based on descriptive clustering), via topics (based on topic modelling), and via similarities to a reference. It then supports screening and prioritises references using relevancy predictions based on an active learning model which learns from reviewers’ screening decisions.

Why is your research important?

RobotAnalyst integrates novel research methods in text mining and machine learning in one system. The methods accelerate screening dramatically. Public health analysts do not have to waste time sifting non-relevant citations within complex reviews.

Our research goes beyond simple keyword extraction, which returns thousands of irrelevant search results due to the amount of the literature. The RobotAnalyst supports human analysts by recording their decisions as they work and learning from them, so that it can apply the same inclusion and exclusion criteria to thousands of documents automatically in a few seconds. It incorporates several text mining and machine learning techniques, including full-text search, multi-word term recognition, topic modelling, descriptive clustering and seriation in an easy-to-use Web interface.

How has your research influenced others?

Text mining and machine learning are important for systematic reviews. Novel methods are required to support every step of evidence synthesis and guideline development. Most offerings are currently proprietary, which does not offer viable solutions to the wider UK academic community. Our work is accessible and interoperable for others.

During the project, we strengthened our contacts with other solution providers, both commercial and academic. We have joined the ICASR collaboration, which aims to promote standards and data re-usability in the field, to make sure that solutions developed independently can be easily combined.

From the start, we cooperated with systematic reviewers developing public health guidelines at NICE. This ensured that RobotAnalyst is both based on the most up-to-date text mining solutions, and is well suited to the needs of end users. They have helped us to appreciate what would be the most important features for them, assisted in development, performed evaluation and suggested future work.

Despite being an ongoing research project, RobotAnalyst is currently used as a screening tool by analysts to develop systematic reviews and guidelines. Specifically, the system has been employed by NICE in developing new guidelines and surveillance reviews on benefits of exercise, behavioural change, fighting addictions, managing weight and others.

The Lausanne University Hospital is currently using RobotAnalyst to investigate best practices in hospital care for older patients. All these efforts were greatly accelerated due to the text mining and machine learning components available in the system, and we expect more benefits for the community to come when the system is fully published and moves into a full product.

The system was presented at Cochrane colloquia and the Global Evidence Summit, where experts from the field gained hands-on experience of it and provided us with valuable feedback. Currently, Cochrane teams are using the system for their systematic reviews.

Thanks to highly active user engagement, we have established collaborations with systematic reviews teams at Cochrane Switzerland; School of Medicine, Kyoto, School of Medicine, Osaka; Health Research Methods, McMaster, all of whom support us in evaluation of RobotAnalyst. Nationally, we work with the COMET initiative and the NC3Rs/CAMARADES team.

How has your research influenced your career?

This research was pivotal for the National Centre for Text Mining. Although we have engaged with systematic review teams in the past (EPPI-centre), it is the first time we had the chance to include research novelty into a prototype system which, even before the end of the project, had gained international appeal among teams nationally (NICE, NIHR, Prospero) and internationally (Canada, Japan, Switzerland, Austria, etc.) It has created tremendous opportunities for continuation of our research and has demonstrated how we can bridge theory with applications for impact.

What has been the role of the MRC-NIHR Methodology Programme?

MRC funding was absolutely crucial for this project. It enabled the text mining and machine learning team to work together with the analysts from NICE from the onset of the project. This enabled a close interaction to support the development of a usable system that saves time and curation effort for the NICE analysts as proven in the thorough evaluation experiments we have jointly conducted.