What’s Known

The best-worst method (BWM) is used across diverse industries to prioritize critical factors. BWM is the state-of-the-art multi-criteria decision-making method.

What’s New

A new set of internal and external factors to screen the performance of Iranian public hospitals is identified using the BWM. External factors (economic and political) have a greater impact on hospital performance than internal factors.

Introduction

Health system performance refers to improving a patient’s overall health while achieving business goals. Health systems are continuously faced with key issues such as inefficiency, the challenge of meeting patient needs, and increased costs. ¹ Improving the performance of hospitals is thus essential to appropriately respond to such challenges and adapt to social changes in modern society. ² To this end, health managers and policymakers need a specific framework to evaluate the performance of hospitals. ¹

Performance evaluation is a key success factor for any organization and a persistent challenge facing managers and stakeholders. ³ Performance evaluation systems allow identification of the current organizational status and quality of activities, especially in complex and dynamic environments. Without such a system, an organization may not survive in the long run. For the first time, in 1859, Florence Nightingale evaluated hospital performance using parameters such as mortality and infection rates. ⁴ However, it is only since 2000 that certain activities have been initiated to develop tools to measure health system performance. ⁵ Key performance indicators (KPIs) are performance-based decision-making tools used by policymakers and managers. ^{6 - 8} These indicators are used to monitor, evaluate, and manage health systems to improve the quality, efficiency, and effectiveness of health care and to plan and develop organizational goals. ⁹

Given the increasing complexity of hospital management, there is a need for tools to evaluate hospital performance based on scientific knowledge. ¹⁰ Such tools must include appropriate indicators that adequately reflect the performance of hospitals. ¹¹ In addition, taking into account possible changes in hospital performance over time, there should be enough flexibility to adapt performance indicators, as new evidence comes to light. ¹² Additionally, it is inevitable that hospitals will have to deal with performance indicators that are at times contradictory, e.g., safety versus satisfaction or cost of goods versus treatment cost. Consequently, it is essential to identify and prioritize indicators in terms of quality and quantity. For this purpose, multi-criteria decision-making (MCDM) is one of the methods used to explicitly evaluate multiple conflicting criteria in decision making. ¹³ Another method is the best-worst method (BWM), developed by Rezaei in 2015. ¹⁴ This method is proposed to solve MCDM problems by comparing the best/worst criterion against all other identified criteria. In comparison with other algorithms, the advantage of BWM is that it requires fewer comparative data and leads to more consistent and stable results. ¹⁵ In addition, it uses pairwise comparison and thus less likelihood of inconsistencies by comparing too many criteria. The results of various case studies that implemented BWM indicated that it outperforms other methods in terms of consistency ratio, minimum violation, total deviation, and conformity. ¹⁴

Although the BWM has been widely used in various industries, its implementation in the health system is scarce. To the best of our knowledge, only a few studies in Iran have implemented the BWM for decision-making. Abadi and colleagues used the BWM to develop a strategic planning framework for expanding the medical tourism industry in Yazd (Iran). ¹⁶ In another study, Rowshan and colleagues used this method to identify and prioritize factors influencing outsourcing in Zeinabieh Hospital in Shiraz (Iran). ¹⁷ In the present study, the BWM is used to identify and prioritize KPIs of public hospitals in Iran.

Materials and Methods

The present study was approved by the Ethics Committee of Shiraz University of Medical Sciences, Shiraz, Iran (IR.SUMS.REC.1396.S274). A cross-sectional study was conducted during 2016-2019 to identify and prioritized KPIs in Iranian public hospitals using the BWM. Initially, the best and worst criteria for decision-making were determined and compared. Then, the problem was optimized, and the weight of each criterion was determined based on two features, namely the probability and the effect. Finally, using the obtained weights, alternatives resulting in a set of criteria were calculated, and the best alternative was selected. We also calculated the inconsistency rate. The BWM consists of six steps:

Step 1: Define a set of decision-making criteria. In this step, a set of indicators required for decision-making is defined as {c₁, c₂, …, c_n}. In addition, factors affecting the performance of public hospitals were extracted based on a previous study. ⁸ Because of the large number of factors and indicators, the content validity ratio (CVR) was used for screening purposes. CVR was quantitatively calculated based on the opinion of an expert panel. The experts were requested to rate each item on a three-point scale, namely “effective”, “effective but not necessary” and “not effective”. Since the expert panel comprised 32 members, CVR>0.29 was considered acceptable. ¹⁸

$\mathrm{CVR}=\frac{\mathrm{NE}-N/2}{N/2}$

where NE: The number of the experts who rated an item as “necessary”, N: The total number of the experts.

Step 2: Define the best (most important) and the worst (least important) criteria. The selection was made by the panel based on their expert judgment.

Step 3: Define the priority of the best criterion among all other criteria by assigning a score from one to nine. The comparison was performed by the experts using the pairwise comparison method. The priority vector of the best indicator compared to others was expressed as A_B=(a_B1, a_B2,…, a_Bn).

Step 4: Define the priority of the worst criterion among all other criteria in the same manner as in step 3. The priority vector of all indicators compared to the worst indicator was expressed as Aw= (a_w1, a_w2, …, a_wn)^T

Step 5: Modeling the research problem with a linear programming formula. In this stage, for each pairwise comparison of the best criterion with the other criteria (W_b/W_j), a technical coefficient of the best comparison (α_bj) was obtained. Similarly, for each pairwise comparison of the worst criterion with the other criteria (W_j/W_w), a technical coefficient of the worst comparison (α_jw) was obtained. These were expressed as:

$w_b/w_j=a_{\mathrm{bj}}$

$w_j/w_w=a_{\mathrm{jw}}$

The maximum difference between the equations was minimized.

$|w_b/w_j-a_{\mathrm{bj}}|$

$|w_j/w_w-a_{\mathrm{jw}}|$

The result of minimizing the equations was the formation of a function with restrictions.

$\min {\max }_j=\left\{\right|w_b/w_j-a_{\mathrm{bj}}|.|w_j/w_w-a_{\mathrm{jw}}\left|\right\}$

such that ${\sum }_j{w}_j=1$, $w_j\ge \mathrm{0;\; for\; all\; j}$.

To solve the equation, a linear programming formula was developed: min ε such that

$|w_b/w_j-a_{\mathrm{bj}}|\le \epsilon .\mathrm{for\; all\; j}$

$|w_j/w_w-a_{\mathrm{jw}}|\le \epsilon .\mathrm{for\; all\; j}$

${\sum }_j{w}_j=1$, $w_j\ge 0.\mathrm{for\; all\; j.}$

Step 6: Solving the linear programming formula to determine the final weights of each criterion. In this stage, indicators were evaluated and prioritized using the obtained weights.

Data were analyzed using LINGO software version 18.0.44 (Lindo Systems, IL, USA).

Results

Based on the opinion of the expert panel, out of 58 external and 33 internal factors affecting the performance of public hospitals, 18 and 11 factors were selected based on the CVR value, respectively. Internal factors were categorized into two dimensions, namely efficiency, and effectiveness. Indicators associated with efficiency were mainly related to financial factors. External factors were categorized into seven dimensions, namely economic, legal, social, environmental, political, technological, and healthcare megatrends. Among these, economic, legal, and political were the most prominent factors negatively affected by budgeting policies and the payment system. A megatrend was also identified in the form of a national health insurance system as well as a shift from employer-based to government-subsidized insurance coverage (table 1).

Among all factors affecting the performance of public hospitals, economic and social dimensions (external factors) received the highest and lowest ranks with the weights of 0.27 and 0.05, respectively. Efficiency and effectiveness (internal factors) were ranked third and fifth, respectively (table 2).

In the “economic factor” category, the highest-ranking indicators were the structure of the payment system, the inflation rate in the health sector, and patient out-of-pocket contribution. Similarly, the hospital budgeting system and mandatory implementation of guidelines were the highest-ranking indicators in the “legal factors” category. In the category of “social factors”, changes in disease patterns and equity in utilization of hospital services were the most important indicators. Of all indicators, privatization in line with Article 44 of the Constitution of the I.R. Iran (political factor) ranked first followed by advancement in health information technology (technological factor) and increased risk of illnesses due to environmental pollution (environmental factor). In the category of “healthcare megatrends”, a paradigm shift from quantity to quality health service provision, national health insurance, and a shift from employer-based to government-subsidized insurance coverage were the most important items. Among all indicators of internal factors, the most important indicators affecting the performance of public hospitals were emergency care waiting time, the cost revenue ratio, the percentage of doctors using guidelines, and the cost-effectiveness ratio of health care services (table 2)

Discussion

The results of the BWM showed that the economic factor (weight: 0.27099) was the most contributing factor affecting the performance of public hospitals in Iran. Among all indicators of the economic factor, the structure of the payment system had the greatest impact on the performance of public hospitals. The payment system has been reported to be one of the major issues, because it is designed to promote competition between hospitals and increase the number of services to attract more patients. ¹⁹ This model is referred to as the pay-for-service approach, whereby payments are made based on the intensity (volume, frequency, duration, and type) of the provided services. In recent years, however, a new megatrend has been observed in which a shift from a pay-for-service to a pay-for-performance model has taken place, whereby higher quality of care has become the basis for the payment system. ^{20 - 22} This is indicative of a paradigm shift from quantity to quality provision of health services. The importance of this paradigm shift was also reflected in our results, as it ranked fifth among all indicators. Implementation of pay-for-performance will eventually lead to the provision of higher quality services and reduced costs. ²³ In addition, one of the consequences of this approach is the development of a value-based purchasing program. ^{22 , 24 , 25} Accordingly, the Iranian health macro policies also emphasize topics such as a quality-based payment system, efficiency improvement, fair salary, and incentives for healthcare providers to improve health care. ²² Unquestionably, hospital performance will markedly improve if healthcare managers and policymakers pro-actively reform the current payment system.

The political factor (weight: 0.151799) was also found to be an important criterion affecting the performance of public hospitals. Based on the opinion of the expert panel, privatization (in line with Article 44 of the Constitution of the I.R. Iran) was the key indicator affecting hospital performance. This approach is an essential component of health system reform, ²⁶ as it will improve hospital performance through cost reduction, faster provision of health care services, higher efficiency, improving the skills of health care workers, and direct focus on key hospital processes. ²⁷ The most important factor for the success of this approach is the collaboration between public and private health sectors in synchronizing their services to increase added value. Additionally, streamlining the structure and organization of hospitals and better evaluating the performance of management teams provide greater adaptability. Furthermore, it gives more flexibility and functional autonomy to hospitals due to their financial dependence on the state budget. ^{28 - 30} However, Kavosi and colleagues warned against outsourcing public health services to the private sector, as it involves complex and multi-criteria decisions. ³¹

Among all factors affecting hospital performance, efficiency and effectiveness ranked third and fifth, respectively. The main indicators associated with these factors were emergency care waiting time (total rank: 6) and cost revenue ratio (total rank: 10). The latter is indicative of the impact of financial conditions on efficiency as an internal factor. In line with our findings, Rahimi and colleagues used a balanced scorecard to determine the performance indicators identified in Iranian public hospitals. They concluded that the cost revenue ratio is one of the key performance indicators. They also showed that this ratio was causally related to as many as 16 different indicators (73%). Another study also reported that financial indicators significantly affected hospital performance and had a causal relationship with other performance indicators. ³

The results of our study showed that emergency care waiting time (total rank: 6) had a great impact on the effectiveness of hospital performance. In a meta-analysis study, Fazel Hashemi and colleagues reported that waiting time in the emergency departments of Iranian hospitals is well above national and international standards. ³² The longer the emergency care waiting time, the less efficient the hospital process. ³³ It is therefore essential to address this indicator not only for its positive contribution to effectiveness, but also for its impact on several other hospital performance factors. Rahimi and colleagues showed that emergency care waiting time has a causal relationship with at least eight other indicators, namely cost revenue ratio, the average length of stay, bed occupancy rate, withdrawal of consent for treatment, hospital infection rate, patient satisfaction, and the number of patient complaints. ³ A triage system is an effective method to reduce waiting time in an emergency department. Choi and colleagues investigated the effect of triage rapid initial assessment by a doctor (TRIAD) on waiting time in an emergency department. ³⁴ They reported that the average waiting time and processing time were reduced by 38% and 23%, respectively. Some other studies have applied different techniques to reduce emergency care waiting time. Based on the quality function deployment technique, Rahimi and colleagues proposed a model to improve the quality of emergency services. ³⁵ In another study, Vashi and colleagues applied lean principles to reduce emergency care waiting times and reported a reduction of door-to-doctor time by 12.6 minutes. ³⁶

The main strength of the present study is the inclusion of both internal and external performance factors affecting public hospitals in Iran. This approach differs from previous studies that solely focused on internal factors. The main limitation of the present study was caused by the COVID-19 pandemic, which hindered the data collection process and negatively affected the completion of the BWM questionnaire. It is recommended that further studies apply the improved version of BWM such as rough BWM or fuzzy BWM. This overcomes uncertainty and ambiguity issues associated with the opinion of the panel of experts. Due to the potential causal relationship between indicators and the complex nature of the topic, the use of soft operational research methods is recommended.

Conclusion

The results showed that external factors (economic and political) had a greater impact on the performance of public hospitals in Iran than internal factors (efficiency and effectiveness). In assessing the effectiveness of interventions and the performance of hospitals, these external factors should be considered.

Acknowledgment

The present manuscript is submitted in partial fulfillment of the requirements for the PhD degree of K. Pourmohammadi (research proposal: 95-01-07-13769). The authors express their gratitude to the Research Council of Shiraz University of Medical Sciences (Shiraz, Iran) for their support.

Authors’ Contribution

P.S: Designed the study, finalized the data synthesis and its overall methodology and revised the manuscript critically for important intellectual content; K.P and R.H: Retrieved and interpreted the data and prepared the initial draft of the article; P.B: Contributed to data analysis and revised the article; N.H: Cooperated in data analysis, improved the structure and conclusion of the paper and revised the manuscript critically for important intellectual content. The study was supervised by PS. All authors read and approved the final manuscript and are responsible for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Conflict of Interest

None declared.

References

1. Peng X, Tang X, Chen Y, Zhang JH. Ranking the Healthcare Resource Factors for Public Satisfaction with Health System in China-Based on the Grey Relational Analysis Models. Int J Environ Res Public Health. 2021; 18Publisher Full Text | DOI | PubMed
2. Rajabi F, Esmailzadeh H, Rostamigooran N, Majdzadeh R, Doshmangir L. Future of health care delivery in iran, opportunities and threats. Iran J Public Health. 2013; 42:23-30. Publisher Full Text | PubMed
3. Rahimi H, Bahmaei J, Shojaei P, Kavosi Z, Khavasi M. Developing a strategy map to improve public hospitals performance with balanced scorecard and dematel approach. JOURNAL DEMO NEOSCRIBER V3 single. 2018; 19:e64056. DOI
4. Gholamzadeh Nikjoo R, Jabbari Beyrami H, Jannati A, Asghari Jaafarabadi M. Selecting Hospital’s Key Performance Indicators, Using Analytic Hierarchy Process Technique. Journal of Community Health Research. 2013; 2:30-8.
5. van den Berg MJ, Kringos DS, Marks LK, Klazinga NS. The Dutch Health Care Performance Report: seven years of health care performance assessment in the Netherlands. Health Res Policy Syst. 2014; 12:1. Publisher Full Text | DOI | PubMed
6. Kazemi M, Karimi MS. The Production Function of the Hospitals in Iran: A Systematic Review. International Journal of Health and Life Sciences. 2020; 6:e102179. DOI
7. Rahimi H, Khammar-nia M, Kavosi Z, Eslahi M. Indicators of hospital performance evaluation: a systematic review. International Journal of Hospital Research. 2014; 3:199-208.
8. Pourmohammadi K, Hatam N, Shojaei P, Bastani P. A comprehensive map of the evidence on the performance evaluation indicators of public hospitals: a scoping study and best fit framework synthesis. Cost Eff Resour Alloc. 2018; 16:64. Publisher Full Text | DOI | PubMed
9. Arah OA, Klazinga NS, Delnoij DM, ten Asbroek AH, Custers T. Conceptual frameworks for health systems performance: a quest for effectiveness, quality, and improvement. Int J Qual Health Care. 2003; 15:377-98. DOI | PubMed
10. Longaray A, Ensslin L, Ensslin S, Alves G, Dutra A, Munhoz P. Using MCDA to evaluate the performance of the logistics process in public hospitals: the case of a Brazilian teaching hospital. International Transactions in Operational Research. 2018; 25:133-56. DOI
11. Wu HY. Constructing a strategy map for banking institutions with key performance indicators of the balanced scorecard. Eval Program Plann. 2012; 35:303-20. DOI | PubMed
12. Organization WH. Selection of indicators for hospital performance measurement: a report on the 3rd and the 4th workshop, Barcelona, Spain, June and September 2003. WHO Regional Office for Europe: Copenhagen; 2004.
13. Pourmohammadi K, Shojaei P, Rahimi H, Bastani P. Evaluating the health system financing of the Eastern Mediterranean Region (EMR) countries using Grey Relation Analysis and Shannon Entropy. Cost Eff Resour Alloc. 2018; 16:31. Publisher Full Text | DOI | PubMed
14. Rezaei J. Best-worst multi-criteria decision-making method. Omega. 2015; 53:49-57. DOI
15. Rezaei J. Best-worst multi-criteria decision-making method: Some properties and a linear model. Omega. 2016; 64:126-30. DOI
16. Abadi F, Sahebi I, Arab A, Alavi A, Karachi H. Application of best-worst method in evaluation of medical tourism development strategy. Decision Science Letters. 2018; 7:77-86.
17. Rowshan M. Providing an Outsourcing Pattern of Selected Departments of Shiraz zeynabiye Hospital by Using Multi-Criteria Decision-Making Technniques Shiraz University. 2018.
18. Ayre C, Scally AJ. Critical values for Lawshe’s content validity ratio: revisiting the original methods of calculation. Measurement and evaluation in counseling and development. 2014; 47:79-86. DOI
19. Gupta H, Barua MK. Identifying enablers of technological innovation for Indian MSMEs using best–worst multi criteria decision making method. Technological Forecasting and Social Change. 2016; 107:69-79. DOI
20. Ginsburg PB. Fee-for-service will remain a feature of major payment reforms, requiring more changes in Medicare physician payment. Health Aff (Millwood). 2012; 31:1977-83. DOI | PubMed
21. Kruse GB, Polsky D, Stuart EA, Werner RM. The impact of hospital pay-for-performance on hospital and Medicare costs. Health Serv Res. 2012; 47:2118-36. Publisher Full Text | DOI | PubMed
22. Werner RM, Kolstad JT, Stuart EA, Polsky D. The effect of pay-for-performance in hospitals: lessons for quality improvement. Health Aff (Millwood). 2011; 30:690-8. DOI | PubMed
23. Nabipour I. Megatrend analaysis of the health policies of IR Iran. ISMJ. 2014; 17:1007-30.
24. Lois Stauffer M. Transforming Health Care From Volume to Value. AAACN Viewpoint. 2017; 39:10.
25. Bastani P, Samadbeik M, Dinarvand R, Kashefian-Naeeini S, Vatankhah S. Qualitative analysis of national documents on health care services and pharmaceuticals` purchasing challenges: evidence from Iran. BMC Health Serv Res. 2018; 18:410. Publisher Full Text | DOI | PubMed
26. Bastani P, Dinarvand R, SamadBeik M, Pourmohammadi K. Pharmaceutical strategic purchasing requirements in Iran: Price interventions and the related effective factors. J Res Pharm Pract. 2016; 5:35-42. Publisher Full Text | DOI | PubMed
27. Kavousi Z, Setoudehzade F, Kharazmi E, Khabiri R, Ravangard R, Rahimi H. The level of propensity to outsource Study: Based on hospital services features by managers and staff perspective in hospitals Kavousi Z Abstract: of Shiraz University of Medical Science, 2010. Journal of Hospital. 2012; 11:9-18.
28. Siddiqi S, Masud TI, Sabri B. Contracting but not without caution: experience with outsourcing of health services in countries of the Eastern Mediterranean Region. Bull World Health Organ. 2006; 84:867-75. Publisher Full Text | DOI | PubMed
29. Akbulut Y, Terekli G, Yildirim T. Outsourcing in Turkish hospitals: a systematic review. Ankara Sağlık Hizmetleri Dergisi.
30. Sadeghi A, Barati O, Bastani P, Jafari DD, Etemadian M. Experiences of selected countries in the use of public-private partnership in hospital services provision. J Pak Med Assoc. 2016; 66:1401-6. PubMed
31. McIntosh N, Grabowski A, Jack B, Nkabane-Nkholongo EL, Vian T. A Public-Private Partnership Improves Clinical Performance In A Hospital Network In Lesotho. Health Aff (Millwood). 2015; 34:954-62. DOI | PubMed
32. Alfen HW, Kalidindi SN, Ogunlana S, Wang S, Abednego MP, Frank-Jungbecker A, et al. Public-Private Partnership in Infrastructure Development-Case Studies from Asia and Europe. Verlag der Bauhaus-Universität: Weimar; 2009.
33. Kavosi Z, Rahimi H, Khanian S, Farhadi P, Kharazmi E. Factors influencing decision making for healthcare services outsourcing: A review and Delphi study. Med J Islam Repub Iran. 2018; 32:56. Publisher Full Text | DOI | PubMed
34. Si SL, You XY, Liu HC, Huang J. Identifying Key Performance Indicators for Holistic Hospital Management with a Modified DEMATEL Approach. Int J Environ Res Public Health. 2017; 14Publisher Full Text | DOI | PubMed
35. Fazl Hashemi SME, Sarabi Asiabar A, Rezapour A, Azami-Aghdash S, Hosseini Amnab H, Mirabedini SA. Patient waiting time in hospital emergency departments of Iran: A systematic review and meta-analysis. Med J Islam Repub Iran. 2017; 31:79. Publisher Full Text | DOI | PubMed
36. Helbig M, Helbig S, Kahla-Witzsch HA, May A. Quality management: reduction of waiting time and efficiency enhancement in an ENT-university outpatients’ department. BMC Health Serv Res. 2009; 9:21. Publisher Full Text | DOI | PubMed