The envy of the world? A closer look at the Commonwealth Fund healthcare study
25 June 2014
SUGGESTED
Education
Government and Institutions
Economic Theory
11 November 2024
‘The NHS has been declared the world’s best healthcare system by an international panel of experts who rated its care superior to countries which spend far more on health’, a triumphant Guardian reports. So there you finally have it, in black and white: there’s no healthcare like state healthcare. The NHS really is the envy of the world, after all. Or is it?
I got a little sceptical when I saw somebody tweet the following quote from the above article: ‘The only serious black mark against the NHS was its poor record on keeping people alive.’ This is mildly irritating. What would you make of a customer review for a coffee machine on Amazon, which awarded five stars, praised the machine to the skies, and then ended by saying ‘The machine has just one minor downside: it has a poor record on making coffee. But otherwise, it’s fantastic, and highly recommended.’
The Guardian article refers to a study by the Commonwealth Fund (CF), which attempts to measure and rank the performance of healthcare systems in 11 developed countries, according to a range of criteria. What is unusual about the study is that it is mostly based on inputs and procedures, not outcomes. There is nothing wrong with that. The problem with health outcomes is that it is notoriously difficult to work out to what extent they are really attributable to the health system, and to what extent they are attributable to lifestyle, environmental or socioeconomic factors. In an ideal world, we would be able to simulate what would happen to health outcomes in country A if it adopted the health system of country B, but kept everything else equal. Or alternatively, if country A kept its healthcare system, but adopted the lifestyle habits of country B. Since our knowledge of healthcare is not advanced enough to do that, the CF approach is as valid as any, and surely an inside view from patients and doctors has to be a valuable perspective. And yet, one has to read the CF study with some caution.
Firstly, the study is built on a very specific idea of how healthcare ought to be delivered, and compares healthcare procedures as reported by doctors and patients to that benchmark. Deviations are counted as indications of poor healthcare, which is often a leap of faith. To give just one example: One criterion by which the CF study evaluates the safety of healthcare is whether a doctor ‘routinely receives a computerized alert or prompt about a potential problem with drug dose or interaction’. On this measure, the UK performs superbly, while Norway and Switzerland perform poorly. But does this really mean that Norway and Switzerland are unsafe places when it comes to drug prescriptions? Or does it simply mean that in these countries, drug safety issues are handled in other ways, which do not fit the CF’s protocol? We cannot know, because the study does not investigate the relationship between those procedures and the relevant outcomes.
My impression is that the CF approach works best for questions where we can safely assume a strong relationship between inputs and outcomes, and where an input has no obvious substitutes. For example, the study asks patients whether they had met a specialist who had no information about their medical history on hand. It is safe to say that something is going wrong if this happens frequently. How is a specialist supposed to make a sound judgment if they have no knowledge of their patients’ medical record? In this category, the UK comes out on top, and that is encouraging. But few other categories are as clear-cut as this.
Secondly, some questions are designed to favour a single-payer, free-at-the-point-of-use system over systems that make greater use of insurance mechanisms or patient co-payments. One question asks whether a patient has ever had out-of-pocket expenses exceeding $1,000 per year, another whether they had experienced disputes with insurers over the payment of medical bills, and another whether they had been discouraged to seek treatment because of its cost. Of course, British patients are bound to answer such questions in the negative, but that is not comparing like with like. The absence of insurance companies or co-payments does not mean that British patients get any treatment they want for free. It means rationing decisions will be made for them by others (doctors, PCTs, or public bodies like NICE), usually without their knowledge. The CF authors are entitled, of course, to argue that limiting access through rationing is fairer than limiting it according to ability to pay. But they should not pretend that the absence of monetary payments means unimpeded access to medicine. It would have made sense to include a control question like ‘Did you ever wish to prescribe a treatment which would have made medical sense, but which is unavailable in your country, or which is heavily discouraged by clinical guidelines on the grounds of cost?’.
Thirdly, and this may be a minor point, the CF study does not attempt to control for social desirability bias, which can be a problem when sentiments towards healthcare systems differ vastly across countries. In the UK, the NHS is an integral part of the national narrative, and criticism of it is heavily socially discouraged. The social insurance systems of the Netherlands, Switzerland or Germany command no such loyalty. Citizens of these countries may also believe that their healthcare systems are the best in the world, and hold some of the principles underpinning them (especially universal access) sacred. But they distinguish much more clearly between those abstract values, which are non-negotiable, and the way the health system currently works, which is open to debate. Unlike here, criticism of the latter is not interpreted as an attack on the former.
Is this an argument against basing a health system study on the responses of patients and doctors? Probably not, as long as the questions are specific enough. But a lot of the questions are phrased in a fairly general way, leaving respondents with considerable room for interpretation.
Finally, while it is inevitable that the study design reflects value judgments, some judgments are more subjective than others. In the CF study, countries gain points if GPs provide lots of advice on healthy living. But do you really want your doctor to routinely nag you to stop smoking, eat more vegetables and exercise more? Do we want GPs to act like fitness trainers? Maybe some of us do, but it does not automatically make for a good healthcare system.
This is not to say that the CF study does not have its uses. Again, health system performance is notoriously difficult to measure, and it is always easy to criticise any particular specification. But NHS statists should not get too carried away. Before taking this study to be the last word, just do a little thought experiment. Suppose you needed treatment abroad, and had a choice between two countries: One in which measurable clinical outcomes are very good, but patients’ endorsements of the system are lukewarm, and one where clinical outcomes are mediocre to poor, but according to patient surveys, everything is wonderful. I know where I would go.
I got a little sceptical when I saw somebody tweet the following quote from the above article: ‘The only serious black mark against the NHS was its poor record on keeping people alive.’ This is mildly irritating. What would you make of a customer review for a coffee machine on Amazon, which awarded five stars, praised the machine to the skies, and then ended by saying ‘The machine has just one minor downside: it has a poor record on making coffee. But otherwise, it’s fantastic, and highly recommended.’
The Guardian article refers to a study by the Commonwealth Fund (CF), which attempts to measure and rank the performance of healthcare systems in 11 developed countries, according to a range of criteria. What is unusual about the study is that it is mostly based on inputs and procedures, not outcomes. There is nothing wrong with that. The problem with health outcomes is that it is notoriously difficult to work out to what extent they are really attributable to the health system, and to what extent they are attributable to lifestyle, environmental or socioeconomic factors. In an ideal world, we would be able to simulate what would happen to health outcomes in country A if it adopted the health system of country B, but kept everything else equal. Or alternatively, if country A kept its healthcare system, but adopted the lifestyle habits of country B. Since our knowledge of healthcare is not advanced enough to do that, the CF approach is as valid as any, and surely an inside view from patients and doctors has to be a valuable perspective. And yet, one has to read the CF study with some caution.
Firstly, the study is built on a very specific idea of how healthcare ought to be delivered, and compares healthcare procedures as reported by doctors and patients to that benchmark. Deviations are counted as indications of poor healthcare, which is often a leap of faith. To give just one example: One criterion by which the CF study evaluates the safety of healthcare is whether a doctor ‘routinely receives a computerized alert or prompt about a potential problem with drug dose or interaction’. On this measure, the UK performs superbly, while Norway and Switzerland perform poorly. But does this really mean that Norway and Switzerland are unsafe places when it comes to drug prescriptions? Or does it simply mean that in these countries, drug safety issues are handled in other ways, which do not fit the CF’s protocol? We cannot know, because the study does not investigate the relationship between those procedures and the relevant outcomes.
My impression is that the CF approach works best for questions where we can safely assume a strong relationship between inputs and outcomes, and where an input has no obvious substitutes. For example, the study asks patients whether they had met a specialist who had no information about their medical history on hand. It is safe to say that something is going wrong if this happens frequently. How is a specialist supposed to make a sound judgment if they have no knowledge of their patients’ medical record? In this category, the UK comes out on top, and that is encouraging. But few other categories are as clear-cut as this.
Secondly, some questions are designed to favour a single-payer, free-at-the-point-of-use system over systems that make greater use of insurance mechanisms or patient co-payments. One question asks whether a patient has ever had out-of-pocket expenses exceeding $1,000 per year, another whether they had experienced disputes with insurers over the payment of medical bills, and another whether they had been discouraged to seek treatment because of its cost. Of course, British patients are bound to answer such questions in the negative, but that is not comparing like with like. The absence of insurance companies or co-payments does not mean that British patients get any treatment they want for free. It means rationing decisions will be made for them by others (doctors, PCTs, or public bodies like NICE), usually without their knowledge. The CF authors are entitled, of course, to argue that limiting access through rationing is fairer than limiting it according to ability to pay. But they should not pretend that the absence of monetary payments means unimpeded access to medicine. It would have made sense to include a control question like ‘Did you ever wish to prescribe a treatment which would have made medical sense, but which is unavailable in your country, or which is heavily discouraged by clinical guidelines on the grounds of cost?’.
Thirdly, and this may be a minor point, the CF study does not attempt to control for social desirability bias, which can be a problem when sentiments towards healthcare systems differ vastly across countries. In the UK, the NHS is an integral part of the national narrative, and criticism of it is heavily socially discouraged. The social insurance systems of the Netherlands, Switzerland or Germany command no such loyalty. Citizens of these countries may also believe that their healthcare systems are the best in the world, and hold some of the principles underpinning them (especially universal access) sacred. But they distinguish much more clearly between those abstract values, which are non-negotiable, and the way the health system currently works, which is open to debate. Unlike here, criticism of the latter is not interpreted as an attack on the former.
Is this an argument against basing a health system study on the responses of patients and doctors? Probably not, as long as the questions are specific enough. But a lot of the questions are phrased in a fairly general way, leaving respondents with considerable room for interpretation.
Finally, while it is inevitable that the study design reflects value judgments, some judgments are more subjective than others. In the CF study, countries gain points if GPs provide lots of advice on healthy living. But do you really want your doctor to routinely nag you to stop smoking, eat more vegetables and exercise more? Do we want GPs to act like fitness trainers? Maybe some of us do, but it does not automatically make for a good healthcare system.
This is not to say that the CF study does not have its uses. Again, health system performance is notoriously difficult to measure, and it is always easy to criticise any particular specification. But NHS statists should not get too carried away. Before taking this study to be the last word, just do a little thought experiment. Suppose you needed treatment abroad, and had a choice between two countries: One in which measurable clinical outcomes are very good, but patients’ endorsements of the system are lukewarm, and one where clinical outcomes are mediocre to poor, but according to patient surveys, everything is wonderful. I know where I would go.
17 thoughts on “The envy of the world? A closer look at the Commonwealth Fund healthcare study”
Comments are closed.
SHARE
Here’s the best way to evaluate the NHS:
Let people opt out of the system in return for a tax rebate equivalent to their contribution.
If the system is as good as its defenders claim, then only a few eccentrics would take this option.
After all, if the NHS really is the envy of the world, why is there any need to force people to contribute?
Watch this space. I’m in the process of writing a paper on that idea.
and a more primitive version was published here: http://www.adamsmith.org/sites/default/files/images/stories/booth-health.pdf
The Commonwealth Fund report relies on highly subjective views and has no method of cross-country control. Because there is no split between funding and provision of medical service in the UK and there is no competition, then consumers and providers get their information from the same source. If the NHS isn’t going to provide a certain service/treatment then patients may not even be told that such a service could be supplied and even staff may not be aware of other (better) ways of providing it. In other systems, providers have an incentive to tell you about what they could so for you, and how they could do it better, if only the funding organisation will pay up. The latter may produce ‘creative dissatisfaction’, whereas the NHS may produce ‘blissful ignorance’.
…or maybe it is just that NHS is the best healthcare system in the world!
@DAO – more comprehensive and in-depth international comparisons, such as those of the OECD and ECHI rate the NHS very poorly, so it is highly unlikely that the NHS is anywhere near the best in the world. Only the CF report says otherwise and it has huge flaws (and the CEO of the NHS is on its board!)
Yes I agree, all the studies are flawed. People who want to profit from privatisation of the NHS will always concentrate on the negative ones.
Yes, I agree all studies have their flaws. People who stand to profit from privatization of the NHS will tend to focus on the negative ones, but this study clearly shows that there is another story.
@DAO – the OECD analysis is government-funded. There is no privatisation’ agenda. There is no comprehensive analysis anywhere in which the NHS does anything other than poorly compared to other countries.
There is no comprehensive analysis full stop…and the government is not interested in privatisation??
@DAO – The OECD analysis – if you bother to look at it (which you haven’t) – is extremely comprehensive. It is easily the most rigorous international comparison. The OECD is funded by government – not just our government but the governments of all OECD members. It has no privatisation or other agenda. It comes out in favour of no particular model of delivery of medical care (in fact it says that no one model is inherently superior). What it does say it that the UK has one of the least efficient medical systems and that the quality of what it provides is well below average, even though the cost is average. You should try looking at its methodology and findings as your comments are not informed by having actually considered any data.
I’ve seen the OECD report. It doesn’t say that UK system is inefficient or that UK quality is low or that UK cost is average. If you are impressed by lots of data thats fair enough, but overall it is not a very useful report (please see my previous comment).
@Anonymous. The OECD report says PRECISELY that the quality and and quantity of medical services in the UK is below average and that the cost is average. It uses those exact words in its summary of the UK.
See the whole thing here and read the country summary for the UK. http://www.oecd.org/eco/healthcaresystemsefficiencyandpolicysettings.htm
Yes, at the point of need, NHS is free. But in the chain before the final point of need, there is increasing privatisation. Just look up the profits of Agencies that provide Nurses on short term contracts.
Yes, multi-national research and analysis suffer from methodological issues regarding equivalence. Mainly due to cultural issues. So although this maybe flawed, any student of research methodology will tell you all studies have limitations, including the OECD. But given this constraint we can still take its findings towards some evidence of truth. If NHS has least spend per capita, how many more billions can we invest and still remain the least. Plenty!!
Secondly, if borrowing at 0% interest rate is not so bad, why not invest in the NHS. Yes, debt does increase, but unlike households, the state borrows from itself. So we are still paying back World War 1 debt – and how many childrens’ children (generations) is that?
Your blog entry also neglects to mention that the same study found the USA’s insurance-based healthcare system was rated the most expensive, the worst overall, the worst in more categories than any other country’s healthcare system, and rated in the bottom half of half of the categories it wasn’t rated worst in. Surely so thorough a blasting of the insurance-based healthcare system shouldn’t be discounted simply because it runs counter to your organisation’s worldview?
@Brian Wakeling – But many insurance-based systems in many countries are consistently ranked very highly. This suggest that it is not the fact that the US system is insurance-based that is the problem. Indeed, a highly (and artificially) restricted supply of medical-school places, strict licensure and the way that only employer-funded medical insurance is tax-deductible are major issues in the US which inflate costs. In other words, factors that the government controls and the AMA influences.