The envy of the world? A closer look at the Commonwealth Fund healthcare study
I got a little sceptical when I saw somebody tweet the following quote from the above article: ‘The only serious black mark against the NHS was its poor record on keeping people alive.’ This is mildly irritating. What would you make of a customer review for a coffee machine on Amazon, which awarded five stars, praised the machine to the skies, and then ended by saying ‘The machine has just one minor downside: it has a poor record on making coffee. But otherwise, it’s fantastic, and highly recommended.’
The Guardian article refers to a study by the Commonwealth Fund (CF), which attempts to measure and rank the performance of healthcare systems in 11 developed countries, according to a range of criteria. What is unusual about the study is that it is mostly based on inputs and procedures, not outcomes. There is nothing wrong with that. The problem with health outcomes is that it is notoriously difficult to work out to what extent they are really attributable to the health system, and to what extent they are attributable to lifestyle, environmental or socioeconomic factors. In an ideal world, we would be able to simulate what would happen to health outcomes in country A if it adopted the health system of country B, but kept everything else equal. Or alternatively, if country A kept its healthcare system, but adopted the lifestyle habits of country B. Since our knowledge of healthcare is not advanced enough to do that, the CF approach is as valid as any, and surely an inside view from patients and doctors has to be a valuable perspective. And yet, one has to read the CF study with some caution.
Firstly, the study is built on a very specific idea of how healthcare ought to be delivered, and compares healthcare procedures as reported by doctors and patients to that benchmark. Deviations are counted as indications of poor healthcare, which is often a leap of faith. To give just one example: One criterion by which the CF study evaluates the safety of healthcare is whether a doctor ‘routinely receives a computerized alert or prompt about a potential problem with drug dose or interaction’. On this measure, the UK performs superbly, while Norway and Switzerland perform poorly. But does this really mean that Norway and Switzerland are unsafe places when it comes to drug prescriptions? Or does it simply mean that in these countries, drug safety issues are handled in other ways, which do not fit the CF’s protocol? We cannot know, because the study does not investigate the relationship between those procedures and the relevant outcomes.
My impression is that the CF approach works best for questions where we can safely assume a strong relationship between inputs and outcomes, and where an input has no obvious substitutes. For example, the study asks patients whether they had met a specialist who had no information about their medical history on hand. It is safe to say that something is going wrong if this happens frequently. How is a specialist supposed to make a sound judgment if they have no knowledge of their patients’ medical record? In this category, the UK comes out on top, and that is encouraging. But few other categories are as clear-cut as this.
Secondly, some questions are designed to favour a single-payer, free-at-the-point-of-use system over systems that make greater use of insurance mechanisms or patient co-payments. One question asks whether a patient has ever had out-of-pocket expenses exceeding $1,000 per year, another whether they had experienced disputes with insurers over the payment of medical bills, and another whether they had been discouraged to seek treatment because of its cost. Of course, British patients are bound to answer such questions in the negative, but that is not comparing like with like. The absence of insurance companies or co-payments does not mean that British patients get any treatment they want for free. It means rationing decisions will be made for them by others (doctors, PCTs, or public bodies like NICE), usually without their knowledge. The CF authors are entitled, of course, to argue that limiting access through rationing is fairer than limiting it according to ability to pay. But they should not pretend that the absence of monetary payments means unimpeded access to medicine. It would have made sense to include a control question like ‘Did you ever wish to prescribe a treatment which would have made medical sense, but which is unavailable in your country, or which is heavily discouraged by clinical guidelines on the grounds of cost?’.
Thirdly, and this may be a minor point, the CF study does not attempt to control for social desirability bias, which can be a problem when sentiments towards healthcare systems differ vastly across countries. In the UK, the NHS is an integral part of the national narrative, and criticism of it is heavily socially discouraged. The social insurance systems of the Netherlands, Switzerland or Germany command no such loyalty. Citizens of these countries may also believe that their healthcare systems are the best in the world, and hold some of the principles underpinning them (especially universal access) sacred. But they distinguish much more clearly between those abstract values, which are non-negotiable, and the way the health system currently works, which is open to debate. Unlike here, criticism of the latter is not interpreted as an attack on the former.
Is this an argument against basing a health system study on the responses of patients and doctors? Probably not, as long as the questions are specific enough. But a lot of the questions are phrased in a fairly general way, leaving respondents with considerable room for interpretation.
Finally, while it is inevitable that the study design reflects value judgments, some judgments are more subjective than others. In the CF study, countries gain points if GPs provide lots of advice on healthy living. But do you really want your doctor to routinely nag you to stop smoking, eat more vegetables and exercise more? Do we want GPs to act like fitness trainers? Maybe some of us do, but it does not automatically make for a good healthcare system.
This is not to say that the CF study does not have its uses. Again, health system performance is notoriously difficult to measure, and it is always easy to criticise any particular specification. But NHS statists should not get too carried away. Before taking this study to be the last word, just do a little thought experiment. Suppose you needed treatment abroad, and had a choice between two countries: One in which measurable clinical outcomes are very good, but patients’ endorsements of the system are lukewarm, and one where clinical outcomes are mediocre to poor, but according to patient surveys, everything is wonderful. I know where I would go.