Multi-criteria decision analysis (MCDA) and its alternatives in health technology assessment


Name Affiliation
Pavel Vorobiev
Russian Society for Pharmacoeconomics and Outcomes Research
Malwina Holownia
Russian Society for Pharmacoeconomics and Outcomes Research
Liubov Krasnova
Russian Society for Pharmacoeconomics and Outcomes Research
contributed: 2015-09-02
final review: 2015-09-02
published: 2015-09-02

Article presents application of multi-criteria decision analysis and its alternatives in Russian Federation. There are presented pros and cons for this type of analysis, as well as their Western (Forsight) and Russian alternatives (rule of square of decision making, P. Vorobiev, 2003). There are also presented stages of MCDA, questions of MCDA and types of MCDA.

Keywords: MCDA, multi-criteria decision analysis, multi-criteria decision making, rule of square of decision making

In 2014, national standard GOST R 56044-2014 "Health Technology Assessment" was approved [1]. It is the first document, in which, in details, all the steps and approaches to the right decisions making are described, regarding the implementation of certain techniques in the health care system, both at the level of health management, so and at a medical institution. However, not all modern approaches included in this standard, some of the new technology of decision-making is discussed in this article.

Before you read this article, we must warn you that in it you will find quite a lot of philosophical and economic concepts, those of which we, being in a rush after simple solutions, have abandoned in recent years. Unfortunately – that is bad. Economy is a philosophical concept, although many see it more like bookkeeping. The economy is considered from the standpoint of cost as a system of social relations. It remains to recall that the cost - according to the classical works of Karl Marx – is the added utility of the product or service created by human work.

Word of the economy in the modern language has many meanings, and at least three of them should be mentioned:

• a set of relations connected with the economic activities of people: the market economy (which is hard to impose in Russia) and the administrative-command (which continues to rule the score in Russia, despite the fact that its place remains in the distant past), traditional (based on simple relations goods for money), socialist (when the human needs are placed in the center, not the problem goods for money) and mixed, which, in fact, is being developed and today in all the leading economies of the world.

• agriculture of particular region, country, group of countries, the individual associations of the countries (BRICS, the European Union) and the world at all large.

• scientific discipline that studies the economic life of the company or its segment.

The most common is the following definition: economy (from the Greek οἶκος - house, farm) is the rules of farming [2]. Economy is the activity, but also a set of relations in the system of production, distribution, exchange and consumption of goods and services. Economics society is complex and all-encompassing factor that ensures the livelihoods of every individual and society as a whole. For experts in the field of clinical and economic analysis, it is important that any economy is a complex system that operates in a high degree of uncertainty, and that the medical component multiplies the degree of uncertainty.

Today in the economy there are two approaches opposed - liberal and socialist. Liberalism - the desire for complete freedom of the individual - means that a person does everything by himself and society only helps him in this. Individual citizens can unite to achieve their goals, and the goals of each one of them may be different. Liberalization of the economy means that all subjects have complete freedom to buy and sell goods and services at any price, which will be willing to buy it, everyone is free to produce, sell and buy everything that can in principle be produced and sold, is not prohibited or doesn’t bring harm to others. Access to all spheres of activity is open to all on an equal footing and a special law (in Russia - the antimonopoly law) can stop individuals or groups of individuals to limit this access and freedom.

Let us return to uncertainty. The world exposed to contingency: it is stochastic. Vibrations of atoms, electrons, positrons are random, but they form an ordered pattern of molecules, which are in Brownian (random) motion. Randomly moving molecules form of matter - gas, liquid, solid objects. These chaotic motions are subjects to certain laws of physics, without these random fluctuations nor alive, neither living matter cannot exist.

Liberalism gives priority to the person, knowing that his actions only at first glance seem random. Set of countervailing actions are driven by progress. If stochastic processes, vibrations and collisions will be organized - everything stops and collapses. The situation of chaos turn into a void - in emptiness there is no movement. Lack of movement is nothing, it is vacuum.

Attempts to organize stochastic processes in society and fluctuations of individuals exist, and will exist. The desire to organize economic relations has taken shape during socialism, which puts the public good at the forefront: the interests of the group, of the society is above all implies the union of the overall effort to achieve common benefits. Even at the expense of the benefits of individuals. The society (or the state, standing for the aspirations of society) forms goals and objectives, plans activities and creates the conditions for their implementation. Nowadays, health care is now completely socialistic, despite the existence in it of germs of market (liberal) relations.

We must not forget that socialism, by reduction to an absurd (it is only at first glance, in fact it is an objective development) always turns into totalitarianism. The question boils down to the limits of power, which depends on the decision of the distinction between liberal and totalitarian systems: the government identifies all the interests of individuals or identifies only some things, especially, formulating rules, guidelines, which regulate the economic behavior of individuals. In other words, it is a dictatorship of the law (the liberal model) - the rules are clearly defined and relatively unchanged or dictatorship of the group to change the law under the circumstances (the socialist model), so the rules change during the game.

It seems that the state should guide public consent or consensus, if consensus on the specific circumstances exists in the society. If there is no consensus - work on specific projects should not be performed! In any society, there are different interests that conflict with each other, which must be agreed, reaching consensus. These interests are expressed in terms of a large number of criteria can be quantified, for example, in monetary criteria, and exist on a qualitative level (related to quality of life, for example). Consequently, for the achievement of consensus must be taken into account these multicriteria differences.

The overall objectives of determining the trend of social (socialist) building, are called the "public good", "common interest". But is the most good and honest government able to cover all the infinite variety of human needs, competing in meeting their needs, determine the weight of each of them on a common (not abstract, but quite specific) scale. How far the multi-criteria approach to the assessment of the situation, that is related to the large number of uncertainties is adequate.

Human happiness (we equal it with satisfaction of all his needs) depends on a variety of reasons, countless combinations. Therefore, the government, ideally, should build just a hierarchy of objectives, the set of admissible models, algorithms in which every person will be able to find a place for satisfaction with each of their needs.

We grow from a planned economy - the highest achievement of the socialist state, in which most of us were born and raised. But in the process of planning is basically impossible to take into account the tendency of individuals and because of it a particular person in a planned economy acts as a tool used by the state to serve the "common good" is considered as a group of individuals, "certain categories of citizens".

What to do? Where strive to invest? Is it better to solve the problem of "socially significant" mass diseases or direct efforts towards rare diseases? And here again comes to the aid the economy: economic theory on the one hand trying to explain social phenomena, mechanisms of interactions in the society, on the other - makes it possible to anticipate trends in the economic behavior of individuals and society as a whole. The most important function of the economy is the scientific prediction of the future.

Now there is a new economic term Forsight - methodology of systematic evaluation of the future of science, technology, economy and society in order to identify areas of strategic research and implementation of technologies that bring the largest economic and social benefits.


Technologies, used in Forsight [3]

  • Backcasting        
  • Bibliometical analysis
  • Brainstorming
  • Citizens Panels
  • Cross-Impact Analysis
  • Delphi       
  • Environmental Scanning
  • Essays
  • Expert Panels
  • Futures Workshops
  • Gaming
  • Key Technologies        
  • Literature Review
  • Megatrend Analysis
  • Modelling and simulation
  • Multi-criteria Analysis
  • Scenarios
  • Stakeholder Mapping
  • SWOT Analysis
  • Technology Roadmapping    
  • Trend Extrapolation

As it is shown in the scheme, multi-criteria analysis is only one of the methods in Forsight methodology. Multicriteria analysis (Multi-Criteria Analysis - MCA) is defined as the structuring process of evaluation and selection of alternatives under conditions of high uncertainty and a conflict of different social groups on the basis of a combination of quantitative and qualitative criteria for the evaluation and comparison of technologies in order to achieve mutual understanding and resolve conflicts between various stakeholders involved in the decision-making process.

Let us explain. It is obvious that in the process of decision-making in health care there are conflicts between providers and producers, consumers and payers, regulators. Of course, you can just sit down at the negotiating table, but obviously it is not always possible to agree on everything. The proposed method of multicriteria analysis - this is a consensus in the process of consultation, based on scientific, structured approach and compromise in the evaluation of pre-agreed criteria.

Pharmacoeconomics as a special case of the clinical and economic analysis (the latter term was proposed by P. A. Vorobiev in the late 90s) emerged in the late 80s of the XX century as a methodology that complements exclusively clinical and social studies when making decisions regarding the payment for medical technologies in the system of health care (reimbursement). Currently, clinical and economic analysis extended in the direction of decision-making on the various medical technologies - non-drug, organizational and others.

Existing in the methodology of clinical and economic analysis approaches that take into account different points of view (the consumer, the provider of medical services and medicines manufacturer and payer) based on an approach of "economic effectiveness", and allow to compare the incomparable: generic with innovative drugs, different medical technologies (drug and non-drug) used in one disease, technologies for various diseases, the effectiveness of health systems in different countries. However, decisions about funding the one or the other medical technology based only on cost / effectiveness ratio, even in the case where the effectiveness is in the basis of the quality of life is not enough objectively.

The imperfection methodology of clinical and economic analysis was especially vivid manifested in diseases for which a purely economic approach is not applicable: medical technologies used in rare diseases. Taking into account only the incremental coefficient «ICER» (in fact - a profitable investment) in the evaluation of these technologies results in a denial of reimbursement for this medical technology and, the refusal to treat these patients. On the one hand, amount of these patients, with each pathology, is not so big, on the other - the number of rare diseases is quite great and if we sum up the total of the group – there is a few millions of patients in Russia. Moreover, how the question of the treatment or non-treatment of these patients is asked, because of the lack of resources seems blasphemous.

It is believed that the inclusion of the different criteria in deciding is the most close to real life, and structuring this technology allows you to make this approach scientific. Multi-criteria decision analysis in medicine is based on the following key assumptions (hypotheses):

-  Lot of subjective assessments lead to one objective (this position causes justified doubts);

-  Traditional clinical and economic approaches are not adequate enough to evaluate all medical technologies (that is absolutely fair);

-  The main goal of decision-making in health care is ensuring fairness and equality for all patients (or at least the desire for it).

Multi-criteria decision analysis uses the following models [4]:

-   Value measurement model: each of the criteria is defined in the numerical expression, compares the sum of criteria for each technology;

-   Out ranking model: comparison of alternatives with each other for each criterion, amount of superiority by rating;

-   Goal, aspiration, or reference-level model: for each of the criteria is determined the level below which model does not work, then there is comparison of the    results of the application of each technology, calculated amount of remaining values.

These approaches are largely mechanistic, based on purely arithmetical values. However, not always simple mathematical formulas play a role in decision-making. In 2003 P. A. Vorobiev proposed an original model of multicriteria analysis – rule of square of decision making [5,6,7]. The essence of the model: to the decision maker, several factors have an impact - information, mathematical modeling, non-specific effects and resistance. The decision maker consistently analyzes these streams: selects from the entire array of information the relevant one, using mathematical apparat or logic models. In the case of health technology assessment results of clinical studies are mathematically estimated (meta-analyzes, systematic reviews), the models of decision making are built (Markov analysis, decision tree), simulation are being conducted.

Most often, this completes the decision-making process, and it is not optimal, since it ignores many other factors that may be important for the realization of what seems to be the right decision.

The main thing - is the impact of non-specific factors that are not captured when the model is being created. A simple example: you work on your computer and suddenly, in the middle of the biggest calculations, the electricity turns off. Not only is the result lost but also the background. In more complex cases, we can talk about the output of any administrative document, crisis, change in the political arena, the emergence of a new manager, natural and other disasters.

In the preparation of civil contracts, these factors are referred to as "force majeure", which means that these circumstances are exempt from the obligations of the contracting parties. However, as a rule, any obligations remain and have to be fulfilled by the contracting parties.

In the decision-making process, all non-specific factors can be identified and the probability of their influence can be assessed with the help of the refined model. Moreover, it is possible that offsetting of two or more non-specific factors (on the one side… on the other side...) or vice versa, a synergistic interaction of seemingly minor factors.

When taking decisions it must be remembered that any decision causes resistance of whom it is directed. A classic example - compliance with type 2 diabetes and hypertension: patients do not take drugs, not only because they forget, but also because they feel that there is no need for lifelong treatment. And in this situation neither they cannot be propagandized, nor, especially, made to.

Considering the fourth verge - verge of the subject-object relationship it is necessary to understand that health care providers, payers, consumers and regulators of health care can act as subjects and objects of the decisions taken. The relationship between subject and object always have the character of conflict. Conflict can be resolved by direct force (then the decision will not be enforceable if it remains disagreement, the decision only exacerbates resistance), but it can also be solved by finding resonance: push, release and push again to reach a consensus [8].




 Picture 1. Rule of square of decision-making

As other decision methodology, multi-criteria analysis is performed stepwise [9]:

  1. Identification of the decision situation.
  2. Choice of the purpose of this decision.
  3. Determination of criteria on the basis of detailed objectives, as well as the minimum level that will satisfy the decision maker.
  4. Assigning weights to each criterion, depending on the preferences of the decision maker.
  5. Getting information about the parameters for decision-making and the development of alternatives.
  6. Evaluation of different options in accordance with established criteria.
  7. Summing the products of different weights of criteria and sub-evaluations of individual options and assess the usefulness of different options.  Option that receives the maximum number of points is the most rational option.
  8. To confirm your choice, you can navigate through the various results of simulations.

Several of listed stages of multicriteria analysis cause questions. For example, the allocation of significant factors or criteria for the assessment. Any technology can be characterized by a huge set of criteria and which of them is a priority and is the most important – that is the big question. Of course, you can organize "rating voting" among of the experts and find out which, in their view, is an important criterion. However, we must not forget about the differences in the angles of various experts - payers, providers and consumers of health services. For some important criteria will be related to the budget, for others - to clinical efficacy, whereas for the third – to the quality of life. However, there may be criteria associated with an increase of life expectancy, which present an abstract concern to society as a whole and they are not as significant from the standpoint of, for example, the payer (short money). 

For example, the criteria proposed for consideration in the evaluation of orphan technology by S. K. Zyryanov [10]:

  1. Availability of affordable and effective drug therapy of specific disease.
  2. Impact on survival prognosis of used therapy
  3. Relationship of symptoms and quality of life.
  4. The impact of the disease on livelihoods, reproduction, professional responsibilities and lifestyle.
  5. The therapeutic effectiveness of innovative medicines.
  6. The effect of a new drug on the prognosis of survival.
  7. Safety the new drug.
  8. Effect of an innovative medical technology on quality of life.

Therefore, even the selection criteria must include mechanisms for reaching consensus. Possible solution of this contradiction is the creation of an open list of criteria for defining to a greater or a lesser extent all medical technologies. This approach seems to be very complex and too little implemented.

The second issue is the formation of focus groups. Clearly, they should represent the various positions of the angles specified above. However, what are the requirements for members of the focus groups? Should it be persons who are far from decision-making procedures or, on the contrary, the persons immersed in this process in one way or another? From a particular solution of this problem can vary quite substantially the result of the evaluation. Selection of experts is arbitrary; we can assume that it is not difficult to find the "right" expert. In addition, experts are subject to the influence of other, external factors: personal experience, ethical preferences, the media, and the environment - it is all taken into account when forming focus groups?

In such a way, the skeptical view of the MCDA procedure allows us to formulate a few key questions to this procedure, while not denying the whole of its positive values and roles.

  1. Who determines the criteria and who evaluates their weight?
  2. How are focus groups created and who is in them?
  3. How to compare different medical technologies with each other?
  4.  Should only the principles of justice and equality stand at the forefront of decision-making?
  5. What should I do if there is unavoidable contradictions?

By a cursory acquaintance with the methodology of multicriteria analysis in decision-making in the first place - on reimbursement - it seems that the multicriteria analysis solves a problem of the verge of subject-object interactions, when seen through the prism of the "rules of the square." At the same time, all other verges "square of decision-making" remains outside of the purview of this type of analysis. Multicriteria analysis complicates the already difficult task of decision-making under conditions of high uncertainty. However, we must not forget about the rule of "Occam's Razor": "Do not attract new entity unless it is absolutely necessary." "Keep it simple" - we say. It should be remembered that the simpler analysis is, the easier it is to play. Don’t we leave this principle by introducing a multi-criteria analysis?

It is enough to recall the Resolution of the Government on August 28, 2014 N 871 "On approval of rules of formation of lists of drugs for medical use and a minimum range of medicines needed for health care" [11], which, though not fully, but is based on principles of multicriteria analysis. Among the criteria, there is the cost of treatment, which, in fact, should not be taken into account when making decisions. Life has shown shortcomings of this approach, when a full-time senior staff of the Ministry of Health discussed (with the whole process live on the Internet), which drugs should be included in the List, and which should not. The whole, formed mathematical base of surrogate multicriteria analysis was not considered.

At the Congress of the International Society of Pharmacoeconomics and Outcomes Research (ISPOR) in Amsterdam (November 2014), a joint meeting of representatives of pharmacoeconomic societies of Central and Eastern Europe on multicriteria analysis in decision making was held. Professor Z. Kalo (Hungary) suggested that this analysis should be a tool, not the rule. By using multi-criteria decision analysis, it is necessary to choose between the two approaches [12]:

1) In addition to the approach of "cost-effectiveness", which is based on existing criteria and processes in agencies for Health Technology Assessment. This reduces the role of the QALY and ICER when making decisions, and additional criteria are taken into account. With this approach, there is more transparency and reproducibility in the decision-making process.

2) Pure, when multicriteria analysis is the sole basis of decision-making.

Thus, multi-criteria analysis should be in some form built into the system of decision-making, as the jury is built into court: their role is to say whether the one is guilty or not, and the role of experts of multicriteria analysis – whether technology is acceptable or not. In our view, multi-criteria analysis should be taken into account in the assessment of medical technology and is complementary to other approaches of decision-making, but cannot be decisive.

  1. ГОСТ Р 56044-2014 "Оценка медицинских технологий"; ​​Available from:; [Accessed: 12.03.2015]
  2. Available from:; [Accessed: 12.03.2015]
  3. Thokala P., Duenas A. Multiple Criteria Decision Analysis for Health Technology Assessment. Value in Health 15(2012); 1172–1181
  4. П.А. Воробьев, М.В. Сура Принятие решений – на пути от теории к практике Проблемы стандартизации в здравоохранении № 6 2004 c. 3-11
  5. П.А. Воробьев, М.В. Сура Развитие теории принятия решений на примере деятельности Формулярного комитета по пересмотру перечня жизненно необходимых и важнейших лекарственных средств (ПЖВНЛС), Проблемы стандартизации в здравоохранении 2004 № 7, с 63-66
  6. П.А. Воробьев Моделирование в клинико-экономическом анализе Здравоохранение Дальнего Востока № 6 (14) 2004, с. 55-61
  7. Vorobiev P., Bezmelnitsyna L., Holownia M. The organization of the health care system in the Russian Federation Journal of Health Policy Outcomes Research, №2, 2012, p.6-10
  8. Walentynowicz P., Jankowska-Mihułowicz M. Wykorzystanie analizy wielokryterialnej w podejmowaniu decyzji kierowniczych, w przedsiębiorstwach województwa pomorskiego;  Available from:; [Accessed: 12.03.2015]
  9. Зырянов С.К., 2014; Available from:; [Accessed: 12.03.2015]
  10. Available from:; [Accessed: 12.03.2015]
  11.  Value assessment methods and application of Multiple Criteria Decision Analysis for HTA. Aris Angelis and Panos Kanavos, Medical Technology Research Group, LSE Health, Advance HTA Capacity Building, September 2014

About Us

Journal of Health Policy & Outcomes Research (JHPOR) is a peer-reviewed, international scientific journal, covering health policy, pharmacoeconomics and outcomes research in Poland and worldwide. The journal is issued under the auspices of the Polish Society of Pharmacoeconomics.

Subscribe to our newsletter:

Latest Articles

Our Contacts

Śliska 3 lok. 55
00-127 Warszawa
NIP 5252390463
REGON 140936540
KRS 0000277843

2017 © Pro Medicina Foundation