Our Methodology

FutureProofing Healthcare is an initiative that aims to accelerate and support the future evolution of healthcare by tracking and measuring progress towards more sustainable, personalised, integrated, and digital health systems. The Indices aggregate accredited, current third-party data to enable us to take stock, learn from best practices and most importantly, start a conversation to shape sustainable healthcare systems for the future.

The indices each reflect the collective thinking of a steering panel of experts, which guided the full index development process. Each of these groups aimed to create something as neutral, relevant and comprehensive as possible, so that we can make existing data work to drive decision making. There are limits to any index, and everyone involved in this effort acknowledges those limits, including: the limited sources of data available, the ability to adequately draw on the collective understanding of as many disciplines and experiences as possible, and the limits of what can be measured.

Data selection process

While the overall dataset is incredibly large, there was a very rigorous process to decide which measures would be contained in it. This process started with an extensive scan to identify possible data sources.

Because all data included in the Index is secondary data, it reflects many different methods of data collection. Given the challenges in cataloguing data across more than 50 countries, there will undoubtedly be debates over the patterns shown in some measures. Naturally, questions may be raised about individual measures – whether they paint an accurate picture of the strength of a healthcare system, whether they can be trusted as they are reported, or if they are the most appropriate way to address a specific issue. Understanding that such debate is a positive development that could drive improvement in future data collection, our approach has been to include as many measures as possible, with the view that the comprehensive scale of the data repository will enable a more complete picture.

Data selection criteria

At the very start of the process, the FutureProofing Healthcare Expert Panel established several criteria that a measure should meet, before being considered for inclusion: 

• Coverage (all or most Member States included)
• Convertibility (ability to be rescaled)
• Trackability (time series, or likelihood of being measured again in future)
• Relevance (to a Vital Sign)
• Credibility (source and method)

Many measures were considered, but not included, as they were judged by the expert panel not to have met the criteria above. 

Of these criteria, three are answered in an entirely objective manner (Coverage, Convertibility, Trackability). The remaining two (Relevance, Credibility) were assessed by the Expert Panel. Here it is worth emphasising again the wide range of perspectives and high level of expertise represented in this panel: this ensured a genuinely expert assessment, and that no single perspective of ‘sustainability’ was allowed to dominate.

More than this, measures were heavily scrutinised by the expert panel, who brought their in-depth knowledge of the subject matter to both assess potential measures and suggest additional data for inclusion.

Complementing our data sources

External Questionnaire

While publicly available data from secondary sources were able to cover many of the concepts addressed in the Personalised Health Index, there were several instances in which data were either too old to accurately reflect the state of personalised healthcare in the examined countries and territories, or simply non-existent. To collect missing data, FutureProofing Healthcare, led by the Copenhagen Institute for Futures Studies, constructed a questionnaire covering several topic areas:

• The existence and level of implementation of electronic health record policies or strategies
• The existence and level of implementation of personalised healthcare policies or strategies
• The alignment of the national healthcare system with value-based care models
• The availability of education and training in personalised health-related fields for the healthcare workforce
• The use and accessibility of health and health-related to patients and for research
• The use of evidence-based guidelines in healthcare
• The existence and level of implementation of policies or strategies for artificial intelligence
• The existence and level of implementation of policies or strategies for genomics and other “-omics”

The questions were primarily modelled on the design of two surveys previously conducted by the World Health Organisation and the Economist Intelligence Unit. All questions featured a multiple-choice component as well as a “Comments” field in which respondents could provide an explanation for or supplemental information related to each response.

An online version of the questionnaire was distributed to officials in national ministries of health (or equivalent public institutions) who were able to answer questions on the above topics in an official capacity. The aim here was to ensure that actors with close and frequent interactions with health policy could provide the most accurate, up-to-date insights as possible concerning the status of personalised healthcare in their respective countries or territories. Responses were collected from August 2020 until mid-October 2020.

When the data collection period concluded, the data from the multiple-choice questions were normalised and integrated into the master dataset. The supplemental qualitative data were not factored in during the normalisation process, but may be used for future studies that permit the use of qualitative data.

Data aggregation and normalisation method

The Index is constructed using a ‘two-stage roll-up method’ – individual measures within a Vital Sign are ‘rolled up’ - or averaged - to produce a Vital Sign score. The Vital Signs are then rolled up to produce a single Index score. To do this, measures that were each expressed in a different way were all normalised to a 1 to 10 scale – where 10 represents the best performing country, 1 the worst performing country. A scale of 1 to 10 was chosen rather than 0 to 10 to reflect the fact that even in countries with the lowest scores, relative to other countries, there is some good practice from which we can learn.

These 1 to 10 scores were then converted to 10 to 100, as overall Index scores are shown with a theoretical maximum of 100.

The method of normalisation used takes into account the distribution of countries from the original data source: for example, if more countries are closer to the ‘best’ in the source data, more countries will score closer to 100 than to 10 on the normalised scale. This enables us to highlight areas where there is inequality across the region. This means that the scores give us a comparison between healthcare systems rather than a score against an absolute ideal, so a 100 or a 10 indicates that a country is performing best or worst out of a region’s healthcare systems right now.

Addressing gaps in the data

Given the number of countries and data sources involved, it is remarkable that many measures contain data for all countries in a given index. Nonetheless, in some cases, data was missing in a measure. To deal with this, two important principles were adopted.

• To ensure good coverage across countries, most sources included in the Index cover a majority or all countries. However, the reality of data collection means that some sources have less coverage. As a general rule, we have looked for a minimum of 50% of countries being covered by a source.  In some cases, when a source does not meet this requirement, the experts have nonetheless determined that the data were too valuable to exclude, in which case the data has been included despite the acknowledged gaps.
• Secondly, the approach taken for calculating a figure for the missing country essentially maintains the position of that country within a Vital Sign, as observed for other measures. This means that a country does not ‘gain’ from not having data for a measure where other countries tend to score low, or ‘lose’ in the case of missing data for a measure that tends to score high.

By using this method, a true value for a particular measure is not being estimated, but the Index ensures that missing data does not impact Vital Sign or Index scores by using an ‘inferred’ value that may bear no or limited relation to the actual value for that measure in that country. It is also made clear to users when a figure has been calculated in this way.

All measures are equally weighted

With so many measures included, the question arises: which are more important than others when it comes to system sustainability? There are many ways this question could be answered, from a purely qualitative judgement on the value of different data points to the use of statistical methods. There was a deliberate decision not to take a view on this – to allow users to apply their own value judgements. 

The way this has been done is simple: within any Vital Sign, all data points count equally. Within the overall Index, all Vital Signs count equally.

Are there limitations to the data?

At all stages, rigour has dictated the principles, methodology and procedures adopted. As with all data-based exercises, there are natural limitations of which users should be aware before making their own judgements as to what can be concluded based on data. The major considerations here are:

• An Index collates heterogeneous data sources, and might therefore be less precise than individual, targeted studies. However, our thorough methodology ensures that data selected in solid and tries to address this loss in precision.
• Only existing data sources can be considered. In Expert Panel discussions, it was clear that there are several measures that would be highly informative, but that simply don’t yet exist. This means that there is, to some extent, an attempt to make an assessment of future-readiness based on the present. Hopefully, identifying these gaps is one of the positive outcomes of the Indices, so that the healthcare policy community may more effectively rethink what data is being collected.
• Some measures have incomplete data (see ‘Treating Missing Data’ above).
• For some measures, data was not collected for all countries at the same point in time (for example, some Eurostat measures give the most recent figures for all countries, as reported by national authorities – which may be on different cycles). This means that the comparison between countries may not be entirely like-for-like. To ensure consistency across countries and to allow each country to control the way in which its data is reported, we have used the data as reported by our source, meaning in some cases it will be collected in different years.
• The Indices are looking at information averaged out across a country and may therefore make local-level (within one country) differences less visible. The objective of the Index is to allow for comparison between countries, so national-level governments can learn from each other’s best practices. 
• An Index provides an overview of health systems at one point in time. The COVID-19 pandemic accelerated healthcare system changes that have yet to be effectively measured and quantified, and are therefore not reflected in our Indices at this stage.

As FutureProofing Healthcare is designed to drive a future-focused conversation that will benefit all healthcare actors, we acknowledge the limitations to current understanding and encourage active input on how to improve them. All feedback and critique is welcome, as these will help to improve the Indices and make them more useful for shaping the healthcare systems of the future.