Content

Our Methodology

FutureProofing Healthcare believes we should all have equal access to healthcare. Our Indices aggregate accredited, current third-party data to enable us to take stock, learn from best practices and most importantly, start a conversation to shape sustainable healthcare systems for the future.

A steering panel of healthcare experts has rigorously guided the building of the Indices to make the final tool as neutral, relevant and wide-reaching as possible, and to drive a future-focused conversation. Despite our rigorous methodology, FutureProofing Healthcare unreservedly acknowledges that the Indices are not flawless – for one, because the available data for healthcare systems is itself not consistent.

Our 2018 Sustainability Index provides a holistic snapshot of what healthcare systems across the 28 EU Member States look like today, with the simple objective of starting a collaborative future-focused conversation on the sustainability of healthcare.

Who created FutureProofing Healthcare?

A steering panel of experts in patient care, health policy, sustainable healthcare and health innovation guided the design of the Index by agreeing on the methodology, directing the data gathering, validating the strength of the sources and signing off on the final tool. 

The project has been financially supported by Roche, because as a stakeholder in healthcare systems the company firmly believes we should be working together to develop holistic sustainable systems for patients. From the outset, Roche has purposefully avoided steering the project to promote its point of view on specific topics. Indeed, the breadth, role and scope of the steering panel of experts played a crucial role in ensuring the approach taken was fair, covered the entire healthcare system and was impartial with regards to the data findings.

What does the Healthcare Sustainability Index measure?

The 2018 Sustainability Index is an initiative aimed at highlighting best practices across EU Member States, raising ‘big questions’ around the sustainability of our systems and facilitating discussion on how to address future challenges and ensure we work towards the future. Its ultimate goal is to contribute to system-enhancement. 

The Index consists of five ‘Vital Signs’: Access, Health Status, Innovation, Quality and Resilience, each consisting of a number of individual measures. 

Why have a Breast Cancer Index?

We are also launching a Breast Cancer Index, which aims to drive discussion around the state of play of this disease across the EU Member States. Through the Breast Cancer Index, FutureProofing Healthcare also takes a closer look at the patient journey through the lens of one disease. This allows us to connect the broader discussion of healthcare system sustainability to actual patient experiences in a very concrete way.

Over the past twenty years breast cancer has made huge strides forward in treatment and care with positive results in outcomes and survivorship. It is a paradigm that should be applied to other areas of healthcare. Breast cancer is also one of the most prevalent cancers with significant data from all 28 EU Member States publicly available, and as a result breast cancer was considered a suitable disease for inclusion in FutureProofing Healthcare.

The construction of the Breast Cancer Index is the same as that of the Sustainability Index, except that the five Vital Signs in this case correspond to the five stages of the patient journey: Prevention and Diagnosis, Treatment, Outcomes and Survivorship, Patient Focus and Palliative Care.

How complete a picture does FutureProofing Healthcare provide? 

A unique strength of the Index is the volume of data it includes across the Healthcare system. In total over 80 Measures and Vital Signs, and more than 2,400 data points, all signed-off by the panel of experts. This depth of data allows for a wide breadth in being able to consider multiple aspects of system sustainability. 

The sheer volume of data also means that each of these fundamental aspects can be looked at from various angles. Each Vital Sign alone is comprised of enough different measures to stand on its own right as a valid index. Together, they represent an incredibly powerful tool for exploring many different angles on healthcare sustainability. 

Where does the data come from? 

All of the data within FutureProofing Healthcare comes from publicly-available third-party sources, which have been validated by the panel of experts as credible sources of reliable data on healthcare system performance, and aggregated into a comprehensive resource on healthcare system sustainability. 

Sources include: the World Bank, UNICEF, the European Environment Agency, Eurostat, the OECD, the European Centre for Disease Prevention and Control, the European Patent Office, the World Economic Forum and the European Commission Initiative on Breast Cancer. Beyond these institutional sources, data from a number of leading academic publications have also been considered. For a list of all the sources used, see Data Sources.

By casting the net far and wide in this way, the resulting public, interactive open source digital platform is a comprehensive resource for taking a data-led approach to drive a dialogue on healthcare system sustainability. Indeed, a large part of the value of FutureProofing Healthcare lies in the fact that it functions as a data repository for anybody to easily explore and analyse the data.

How does one interpret the data?

Because all data included in the Index is secondary data, it reflects many different methods of data collection. Given the challenges in cataloguing data across 28 Member States, there will undoubtedly be debate over the patterns shown in some measures. Naturally, questions may be raised about individual measures – whether they paint an accurate picture of the strength of a healthcare system, whether they can be trusted as they are reported, or if they are the most appropriate way to address a specific issue. Understanding that such debate is a positive development that could drive improvement in future data collection, our approach has been to include as many measures as possible, with the view that the comprehensive scale of the data repository will enable a more complete picture.

How was the data selected?

While the overall dataset is incredibly large, there was a very rigorous process to decide which measures would be contained in it. This process started with an extensive scan to identify possible data sources.

At the very start of the process, the FutureProofing Healthcare Expert Panel established several criteria that a measure should meet, before being considered for inclusion: 

  • Coverage (all or most Member States included)
  • Convertibility (ability to be rescaled)
  • Trackability (time series, or likelihood of being measured again in future)
  • Relevance (to a Vital Sign)
  • Credibility (source and method)

Many measures were considered, but not included, as they were judged by the expert panel not to have met the criteria above. Of these criteria, three are answered in an entirely objective manner (Coverage, Convertibility, Trackability). The remaining two (Relevance, Credibility) were assessed by the Expert Panel. Here it is worth emphasising again the wide range of perspectives and high level of expertise represented in this panel: this ensured a genuinely expert assessment, and that no single perspective of ‘sustainability’ was allowed to dominate.

More than this, measures were heavily scrutinised by the expert panel, who brought their in-depth knowledge of the subject matter to both assess potential measures and suggest additional data for inclusion.

How was the data aggregated and normalised?

The Index is constructed using a ‘two-stage roll-up method’ – individual measures within a Vital Sign are ‘rolled up’ - or averaged - to produce a Vital Sign score. The five Vital Signs are then rolled up to produce a single Index score. To do this, measures that were each expressed in a different way were all normalised to a 1 to 10 scale – where 10 represents the best performing country, 1 the worst performing country. A scale of 1 to 10 was chosen rather than 0 to 10 to reflect the fact that even in countries with the lowest scores, relative to other countries, there is some good practice from which we can learn.

The method of normalisation used takes into account the distribution of countries from the original data source: for example, if more countries are closer to the ‘best’ in the source data, more countries will score closer to 10 than to 1 on the normalised scale. This enables us to highlight areas where there is inequality across the EU. This means that the scores give us a comparison between healthcare systems rather than a score against an absolute ideal, so a 10 or a 1 indicates that a country is performing best or worst out of European healthcare systems right now.

How did you address gaps in the data?

Given the number of countries and data sources involved, it is remarkable that many measures contain data for all 28 EU Member States. Nonetheless, in some cases, data was missing in a measure. To deal with this, two important principles were adopted.

  • Firstly, it was vital to limit the number of countries for which data could be missing in order to be included in the Index. The standard limit was no more than five countries with missing data. However, this led to gaps for certain types of data that the experts felt were too valuable to exclude, resulting in the inclusion of six measures with greater than five countries missing, all of which were considered essential to create a complete picture of healthcare sustainability.
  • Secondly, the approach taken for calculating a figure for the missing country essentially maintains the position of that country within a Vital Sign, as observed for other measures. This means that a country does not ‘gain’ from not having data for a measure where other countries tend to score low, or ‘lose’ in the case of missing data for a measure that tends to score high.

By using this method, data is not being ‘inferred’ or a particular measure being estimated, but the Index ensures that missing data does not impact Vital Sign or Index scores. It is also made clear to users when a figure has been calculated in this way.

 
Are some measures more important than others?

With so many measures included, the question arises: which are more important than others when it comes to system sustainability? There are many ways this question could be answered, from a purely qualitative judgement on the value of different data points to the use of statistical methods. There was a deliberate decision not to take a view on this – to allow users to apply their own value judgements. 

The way this has been done is simple: within any Vital Sign, all data points count equally. Within the overall Index, all Vital Signs count equally.

How can the data be used?

One reason for being agnostic on the weight given to individual data points is the fact that there is a strong desire to put the data firmly in the hands of the user. The Index digital platform allows users to examine the measures or Vital Signs that matter most to them, and to explore their own hypotheses on the relationships between them and then share and comment on these if they like.

This has also been done to maximise transparency. In any exercise of this nature, on any topic, stakeholders will want to be able to break Index scores down and understand how they are constructed. As well as making all data available to users on the platform, this also extends to linking to original data sources: in case any user prefers to consult the original source.

Are there limitations to the data?

At all stages, rigour has dictated the principles, methodology and procedures adopted. As with all data-based exercises, there are natural limitations of which users should be aware before making their own judgements as to what can be concluded based on data. The major considerations here are:

  • Only existing data sources can be considered. In Expert Panel discussions, it was clear that there are several measures that would be highly informative, but that simply don’t yet exist. This means that there is, to some extent, an attempt to make an assessment of future-readiness based on the present. Hopefully, identifying these gaps is one of the positive outcomes of the Index, so that the healthcare policy community may more effectively rethink what data is being collected.
  • Some measures have incomplete data (see ‘Treating Missing Data’ above).
  • For some measures, data was not collected for all countries at the same point in time (for example, some Eurostat measures give the most recent figures for all countries, as reported by national authorities – which may be on different cycles). This means that the comparison between countries may not be entirely like-for-like. To ensure consistency across countries and to allow each country to control the way in which its data is reported, we have used the data as reported by our source, meaning in some cases it will be collected in different years. 

As FutureProofing Healthcare is designed to drive a future-focused conversation that will benefit all healthcare actors, we acknowledge the limitations to current understanding and encourage active input on how to improve them. All feedback and critique is welcome, as these will help to improve the Indices and make them more useful for shaping the healthcare systems of the future.