While there may not be a ‘perfect’ way to rank health care systems, the WHO needs to at least explicitly acknowledge the limitations of their rankings, and inform the reader of the fact that these values and assumptions are not universally held.

In 2000, the World Health Report’s (WHR) Health System rankings unsettled the international health community with confusion and conflict. Forty percent of the Italian population thought there was so much wrong with their health care system that it should be completely redone. However, the WHR had ranked Italy second in the world. The report ranked the United States not first, not even top five, but thirty-seventh for overall health system performance. In the US and around the world, the future of healthcare policy was on the line. Moreover, the message policymakers could take from these results, assuming the results were reliable enough to take any message from them at all, was mired in controversy.

The purpose of the WHR is to provide information about health care and health care systems to policymakers and relevant organizations so they can make appropriate decisions about policy and funding. The explanations for such unexpected results can only be found in an examination of the criteria that were used to determine healthcare systems that ranked highest in “performance.” The WHR used three main indicators: effectiveness, fairness, and responsiveness.

“Effectiveness” was supposed to measure the health care system’s medical competence. However, the WHR’s method of measuring “effectiveness” was based on faulty assumptions. “Effectiveness” was measured in respect to the rate of reduction of a nation’s mortality and morbidity rates, assuming that a nation’s health care system is primarily responsible for reductions in mortality. The WHR states that “how the system responds to health needs…shows up in health outcomes”, but there is little evidence that mortality rates correspond with health care quality. In fact, there is copious evidence that social, political, and economic factors, not medical advances, are responsible for changes in mortality rates. The WHR attributes the drastic declines in mortality rates that occurred in the 20^th century entirely to medical advances, but many of these changes occurred long before medical techniques were effective. Mediterranean nations like Spain and France ranked disproportionately highly in this category because their life expectancies are so high, despite extremely low satisfaction from patients. The WHR states, “If Sweden enjoys better health than Uganda—life expectancy is almost exactly twice as long—it is in large part because it spends exactly 35 times as much per capita in its health systems,” indicating that the WHR assumes that medical resources correspond directly with better health and life expectancy. However, in addition to medical resources, a nation’s birth rate, cultural factors, sanitation, eating habits, and general standard of living contribute to its mortality rate as well. The true causes are rooted in economic prosperity and overall increase in quality of life, outside of any medical lifesaving techniques.

Even if one accepts the WHR’s assumption that death rate is an accurate indicator of medical effectiveness, the statistical methods used to describe a nation’s mortality are controversial and seldom used outside the WHR. The WHR used the Disability Adjusted Life Expectancy (DALE) rate of a country to determine the mortality rate.  The DALE ranks disabilities, and has often been accused of undervaluing the lives of disabled people. The WHR assumed that mortality rates correspond with health care quality, and based their mortality statistics on a controversial model.

The second indicator was “fairness”. Fairness refers to the equality of access to health care resources. The measure of how fair a system was based on the controversial assumption that the proportion of money a person spends on their health care to non-food expenditures should be equal for all citizens of a nation. This system assumes that the rich will spend more and make up for the poor spending less. In this case, the poor would spend very little because they can’t afford medical expenses, and the rich spend exorbitant amounts of money on luxury procedures. Since the proportions would be equal, this system would be deemed “fair” by the WHR. Where and on what sort of health care one’s money is spent is an essential issue that is not addressed in this ranking. Also, this system does not differentiate between racial, ethnic, gender, age, or social status divisions, so while the overall “fairness” of a population could be ranked highly, a small subgroup could be grievously mistreated.

The third indicator the WHR used was “responsiveness,” or a health care system’s ability to protect a person’s dignity, providing prompt care and a choice of provider. This indicator deals most directly with customer satisfaction, and it seems safe to assume that the best indication of how well patients are treated by the system would come from patients. However, the WHO calculated their rankings by hand picking “key informants” who determined the values and weights of the indicators that made up “responsiveness.” Little information on the identity of these “key informants” is given in the report and the fear is that they may have had a disproportional influence on the rankings.  The key informants were pulled from only 35 out of the 191 nations, and over half of them were WHO staff.

The WHO’s “performance” ranking is the sum of three weighted indicators, the effectiveness of the system, the degree of financial fairness, and the responsiveness of the system to the user. These indicators are unconventional and vague to begin with, and weighing and synthesizing them into a single measurement further obfuscates the meaning of the results. Ranking healthcare systems on only one all-encompassing measurement grossly oversimplifies their true complexities.

In addition to the issues with the indicators used, the WHR’s data is incomplete. This forced the WHO to make estimates for much of their data. Usually, when estimates are made, old data is taken into account, and the statistics are thoroughly peer-reviewed by statisticians. Since the indicators and data used here is either seldom used or new altogether, there was no old data to compare. Of the 32 articles that were cited affirming the methodologies in the report, 26 were non-peer reviewed internal WHO documents by the authors of the WHR. Since the nations ranked in this report are from all different cultural and economic histories and situations, it should be necessary to examine improvement over time to assess the “performance” of a health care system. However, the WHR states that “determining how to evaluate progress rather than only a health system’s current performance is one of many challenges for future effort” and it is not really taken into account in the rankings.

While there may not be a ‘perfect’ way to rank health care systems, the WHO needs to at least explicitly acknowledge the limitations of their rankings, and inform the reader of the fact that these values and assumptions are not universally held. The WHR could guide policymaking in nations where promotion of a private provider system could be disastrous. By ranking Colombia’s private insurance system (modeled after the U.S.) highly, the WHR suggests that other Latin American nations should take Colombia as an example. However, in nations where there is a large gap between the upper and lower classes and no robust middle class, a private health care system will benefit the rich and severely restrict the poor’s access to health care.

In the decade since the WHR’s publication, it has both helped and harmed the international community. Citing the rankings without careful analysis of how they were developed has proven extremely misleading, but several big picture ideas from the report have contributed to health care in a positive way. The rankings themselves have generated interest in comparing health care systems, and the debate over the credibility of the WHR’s statistics has stimulated new research and analysis projects to clarify and improve the data. For example, the issue of mortality rates not corresponding to health care quality has been addressed by beginning to collect data on deaths from causes that are preventable with timely and effective medical care.  The WHR has created a discussion of the relationship between the government and health care, and the concept that the government is responsible not only for providing resources, but managing those resources and ensuring results. The WHR continues to inspire discussion about how to improve health care in the U.S. and abroad, despite the limitations of the report itself.