Determining of Factors Influencing the Success and Failure of Hospital Information System and Their Evaluation Methods: A Systematic Review

Background: Nowadays, using new information technology (IT) has provided remarkable opportunities to decrease medical errors, support health care specialist, increase the efficiency and even the quality of patient’s care and safety. Objectives: The purpose of this study was the identification of Hospital Information System (HIS) success and failure factors and the evaluation methods of these factors. This research emphasizes the need to a comprehensive evaluation of HISs which considers a wide range of success and failure factors in these systems. Materials and Methods: We searched for relevant English language studies based on keywords in title and abstract, using PubMed, Ovid Medline (by applying MeSH terms), Scopus, ScienceDirect and Embase (earliest entry to march 17, 2012). Studies which considered success models and success or failure factors, or studied the evaluation models of HISs and the related ones were chosen. Since the studies used in this systematic review were heterogeneous, the combination of extracted data was carried out by using narrative synthesis method. Results: We found 16 articles which required detailed analysis. Finally, the suggested framework includes 12 main factors (functional, organizational, behavioral, cultural, management, technical, strategy, economy, education, legal, ethical and political factors), 67 sub factors, and 33 suggested methods for the evaluation of these sub factors. Conclusions: The results of the present research indicates that the emphasis of the HIS evaluation moves from technical subjects to human and organizational subjects, and from objective to subjective issues. Therefore, this issue entails more familiarity with more qualitative evaluation methods. In most of the reviewed studies, the main focus has been laid on the necessity of using multi-method approaches and combining methods to obtain more comprehensive and useful results.


Background
Nowadays, using new information technology (IT) has provided remarkable opportunities to decrease medical errors, support health care specialist, increase the efficiency and even the quality of patient's care and safety (1,2). On the other hand, there are numerous problems in the scope of IT-based systems in the field of health care; therefore, it causes a deep gap between the positive potential for IT to help health care organizations and their negative impacts. It means that a huge amount of money is invested in health information systems, but a significant portion of this money is wasted for inefficient systems or not implemented ones (3).
Evaluation means "the act of measuring or exploring properties of a health information system (in planning, development, implementation, or operation), the result of which informs a decision to be made concerning that system in a specific context" (4). An evaluation which is carried out based on suitable investment and approved techniques can cause the organization to have a forward movement (5,6). Organizations require a comprehensive evaluation framework, which can help create and develop methods of information system evaluation (7); on the other hand, the identification of methods for information systems evaluation can be possible through the identifying the success and failure factors of these systems (8). When we talk about success, we should identify what the criteria and parameters used for the evaluation of success are. Success is considered as a dynamic concept (9). Whether the system achieves its intended purpose from its establishment is what we mean by success; moreover, it should be carried out based on an anticipated time table and budget, while the project team and its users are satisfied with the results and this satisfaction should be constant (10). Since the information system is complicated and multidimensional, it may succeed or fail in different situations (11,12). These days, due to increasing attention of different organizations to expenses related to the projects of information systems and gaining at least the minimum benefits from them, studying their success or failure has its special importance (13)(14)(15). There are various reports presented pertaining to the high rates of failure in IT projects in industrial sector and health care organizations especially in hospitals (3,(16)(17)(18)(19). Kaplan in his study asserts that the rate of failure in implementation of IT in health care organizations of America is almost 50 percent (20).
Information systems are the combination of different elements, among which the measurement of some elements is easier than that of the others. The evaluation framework which is simply concentrated on elements whose measurement is easier cannot introduce a perfect framework in evaluation of information systems (21). Many case studies have been done, whose results are based on one or a few factors which affect the success of IT-based systems, so it mentions this important point that the findings of such literatures should be put together as a puzzle. In this case, a lot of benefits can be achieved from the significant studies done in this field by using different approaches (22). Just one comprehensive evaluation study can show whether a specific system can be successful in a special place or not. Moreover, each evaluation criterion must be measured through an appropriate method. A perfect evaluation should include all proper success factors (23,24).
Previous researchers have discussed the challenges of the evaluation of health information systems and the problems resulting from lack of unique conceptual framework to guide evaluation researches (25,26). The most significant challenge in evaluation of information systems used in health care is that information systems can influence the improvement of treatment and the patient's health level. Due to this potential influence on the patient's life, more accurate criteria should be applied for the evaluation of these systems (5,11,27). Of course, understanding the effects, results and prerequisites necessary for successful implementation of information technologies in health care require a multi-factor viewpoint (15). Some success and failure factors have either less or more importance in different information systems (28). The main focus of our study is placed on the HIS, a computer system is designed to support the hospital needs for comprehensive information including patient's information, clinical information and financial management (29,30). The main purpose of this system is improvement the quality of care provided for patients (31,32). Although the integrated HIS leads to more efficient use of sources, it has not resulted in effective service offering, improve-ment quality of care and increased productivity in many countries (11,33). Unfortunately, the measurement methodologies of different effects of these systems have not been improved along with the development of these systems, while the future managers and users of HISs need accurate evaluation of these systems (11,34).

Objectives
The purpose of this systematic review was the identification of the HIS success and failure factors and the evaluation methods of these factors. This study emphasizes the need to a comprehensive evaluation of HISs which considers a wide range of success and failure factors in these systems.

Materials and Methods
We searched for relevant English language studies based on keywords in title and abstract, using PubMed, Ovid Medline (by applying MeSH terms), Scopus, Scien-ceDirect and Embase (earliest entry to March 17, 2012). In addition, we applied methods such as documentation review of relevant agencies (like Statistics and Information Technology Office in Ministry of Health and Medical Education) and databases that provide grey literature (like system for information on grey literature) for publication bias control. Also, searching was supplemented by scanning bibliographies from identified studies. The key journals have been manually searched to find references which may not be found in databases and the list of references. Table 1 represents the used search strategy and Figure 1 shows a flowchart representing the search and way of choosing studies.   Two reviewers independently examined all titles and abstracts. Cohen's Kappa coefficient was used to measure the inter-rater agreement on the inclusion or exclusion of the articles (k=0.77). The difference between the ideas of the two reviewers was settled by asking the idea of the third reviewer. To control assessment bias, reviewers were blinded about each other's decisions and some information such as the names of journals and authors that could influence their decisions. Studies which considered models and success or failure factors, or studied the evaluation models of the HIS and the related ones were chosen. The studies examining the information systems in a specific area apart from the field of health care and in the field of public health as well as primary care in addition to non-English articles were excluded from the study.
Assessing the quality of qualitative research has attracted much debate and there is little consensus regarding whether quality can or should be assessed in relation to qualitative research (35)(36)(37). Since most articles which entered in this study were review articles and data analysis was done on the words and phrases in the articles using meta synthesis, to avoid losing valuable insights that existed in some studies that might be omitted in critical appraisal, we did not assess the quality of the selected studies.
Since the studies used in this systematic review were heterogeneous, the combination of extracted data was carried out by using narrative synthesis method. Finally, the common and varying aspects of the factors and their evaluation methods used in the selected studies were determined. Then these factors and methods were classified in different groups, and a framework was offered for the evaluation of success and failure factors in the HIS.

Results
Searching the online databases resulted in 475 articles. Initial screening of titles and abstracts rendered 63 articles eligible for further full-text review. Six additional articles were identified by reviewing the bibliographies, yielding a total of 69 articles. Based on reviewing the fulltext of remaining articles, 53 articles were excluded since they did not correspond with the criteria and purpose of this study, and finally 16 articles remained which required detailed analysis. The summaries of findings pertaining to these articles are presented in Table 2. Most selected studies considered factors such as management (n = 11), behavioural (n = 15), organizational (n = 13) and economy (n = 11) as success factors (Table 3). Moreover, some evaluation methods such as interviews (n = 4), questionnaires (n = 5) and usability measurement methods (n = 7) were the most common methods in selected articles (Table 4). prior negative experiences with Clinical Information System (CIS) projects); Usability (poor perceived system ease of use; poor perceived system usefulness; misalignment of system with local practices and processes); Project team (changes to membership on the project team; poor project leadership; lack of required knowledge or skills; lack of clear role definitions; negative attitude of project team members); Project (large and complex project; project ambiguity; changes to requirements; insufficient resources; lack of a project champion; lack of a formal project management methodology); organizational/ environmental (lack of commitment from upper management; organizational instability; lack of local personnel knowledgeable in IT; legal and ethical constraints; privacy and confidentiality issues); strategic/political (misalignment of actors' and partners' objectives and stakes; political games/conflicts; unreliable external partners) Not mentioned

P. Yu/ 2010 (39)
Literature review and a case study End user HIS perspectives Exploratory study; Feasibility study; Cost-productivity; Risk analysis Success factors including: Role of the hospital management (efficiently utilized skills of IS users; contributed to the IS planning; active participation); Education and training (training provided during the IS introduction; IT department providing active support); User support during the implementation (motivators were offered to the employees; external consultant (for IT department); management provided support to users); Contracted task agreement (contract-maintenance support provided; formal documentation system followed; criteria for system specs specified; revision-modification was agreed; schedule-deliverables agreed); Integrated complete information system; Technical specifications (maintenance -system scalability is ensured; specific use instructions are provided; data back-up mechanisms are supported; data security is adequate; IS is fast and flexible; The menus are localized); Standards and coding (HL7; DICOM; GMDN; ATC; ICD10); Evaluation of IS (patient satisfaction evaluation; employee satisfaction evaluation; measurement of cost-productivity)

MM Yusof et al./ 2008 (41)
Literature review and a case study Technology: System quality (data accuracy; data currency; database contents; ease of use; ease of learning; availability; usefulness of system features and functions; flexibility; reliability; technical support; security; efficiency; resource utilization; response time; turnaround time); Information quality (importance; relevance; usefulness; legibility; format; accuracy; conciseness; completeness; reliability; timeliness; data entry methods); Service quality (quick responsiveness; assurance; empa- Analysis of the conference proceedings and a Delphi study Success factors: Functional (careful preparation of the user requirements specification to appropriate and balanced take into account and express users requirements; needs as well as demands; alignment of the role and design of the IT-system; coping with the complexity; flexibility towards dynamic changes and changes in the organizational context; added functionality are provided by the IT-based system; enabling users to provide new or better services); Organizational (collaboration and cooperation; make implementation a transparent process within the organization; work from the workflow; high competences; support from higher level organizations); Behavioral (the users are key; the personal attitude; engagement and commitment; motivational activities); Cultural (understand medicine and healthcare in general as a separate culture; understand the local culture; preparedness and willingness towards cultural change); Political (high-level commitment; monitoring political implications; considering IT-systems a service rather than a product from a vendor; collaboration in providing new solutions; transparency); Management (management support; flexible planning; prospective and proactive control; consider IT implementation as a change process; coping with the impact of change; user involvement; strategy; communication; handling the diversity within stakeholder goals); Technical (standard based; data validity procedures are part of system qualities; use proven technology; usability; integrated functionality; communication standards; balance between flexibility and stability; evolution rather than revolution; flexibility and adaptability; enabling future functional and technical changes); Legal aspects (know what the legal constraints /opportunities are); Strategy (national; regional; organizational; accepted also at lower levels); Economy (there has to be a return of investment (whether material or immaterial); justification of increase of costs; sufficient funding); Education (sufficient training); User acceptance Not mentioned Failure factors: Functional (the system does not meet expectations; limitations in the way the user can express his/herself; moving target); Organizational (not understanding the organizational context); Behavioral (overloading the user; underestimating user acceptance; resistance because of fear or loss of control of own job situation); Cultural (assuming that what works at one place also works somewhere else; users have too high expectations); Management (overambitious implementation plans; judgment based on wrong premises; improper tendering; business reorganization of the vendor); Technical (limitations in the way the user can express his/herself; the technology is so restricted that it impacts design and implementation choices; response rate and other performance measures; vendor did not support the functionality quoted; insufficient verification of conformity with requirements specification); Legal (low concern on regulations and standards; compliance with laws and existing ethical rules of conduct); Economy (lacking financial power of a vendor); Education (visible discrepancy between successive versions of the IT-based system)

Discussion
The results of the present research indicate that the emphasis of HISs evaluation moves from technical subjects to human and organizational subjects and from objective to subjective issues. Therefore, this issue entails more familiarity with more qualitative evaluation methods.
It has been proved that the application of HISs can create basic changes in culture, policy and authority, which link professional groups to one another in an organization. However, these basic subjects have not been identified in many of success models. These issues result in the inability of these models to interpret some cases of failure (39). Many studies published about the success of information systems have been carried out by focusing on the model of Delone and McLean (48,49). Since in this model some important factors have not been considered, the mentioned studies cannot present a comprehensive model in this field. For instance, they deal with supporting the top managers and the involvement of users, and point out that these factors can influence the level of success, but these variables are not considered in this model. Moreover, other factors like culture and organizational characteristics are not considered in this model. The changes of procedures and culture are among the obstacles which are reported in widespread use of health care information systems (41). Other studies emphasized on organizational and management factors such as management commitment, a champion and his/her characteristics (28,38,41,51,52).
The findings indicate that, among different factors of success, user satisfaction to measure HISs success is of special significance (3,11,15,17,23,28,(38)(39)(40)(41)(42)(43)(44)(45)(46). This factor is probably the most widespread factor used in measurement of success (11,41,42). Even though some researchers may doubt about the reliability and appropriateness of using the user satisfaction to measure the success of information systems (52,53), the researches which have been carried out, have proved that satisfaction is a useful criterion to measure the success of the system (11,32,54,55). However, in the field of health care information systems, only four percent of researches have used the criterion of the influence of user satisfaction, whereas this rate reaches almost 20 percent in the researches of information systems (56). The findings show that, in published literatures, special attention has been paid to the economy factors and their evaluation (3,11,15,17,28,38,40,41,(43)(44)(45)(46). On the other hand, when the efficiency of investment in IT is studied, some problems manifest themselves as follows (11): 1-The expenditures are mostly indirect, so their calculation is difficult 2-The benefits and organizational impacts are often intangible, so their realization may take a long time In spite of these problems, findings show that there are appropriate and verified methods for evaluating economy aspects which provide useful information in this area. The other point which seems noticeable about the success factors of the HIS refers to two of selected articles indicating that the evaluation of HISs is one of the success factors (40,44). We believe that the quality and quantity of evaluations and using their results at different stages of the HIS life cycle can be an influential and significant factor in the success of such systems, and this factor can be studied through methods like interviews and studying the existing documents.
The findings reveal that, among the suggested methods for the evaluation of success or failure factors of HISs, using questionnaires (11,23,39,45,47) and interviews (23,39,45,47) are more emphasized. Questionnaires are the best approved method for the evaluation of personal opinion, perception and attitude which is widely used in information systems and health researches. A comprehensive interview and focus group interview are effective on understanding how and why events have happened or will occur and on the perception of users about how affairs can be done more efficiently (39). Among the evaluation methods extracted for this research, some methods are based on the retrospective nature (like functionality assessment). The other kinds such as balanced scorecard and delphi methods might present a guideline for planning and revision HISs, which provide formative evaluation (57). Some methods may not meet the specific information needs in the related field completely, but can be used as valuable supportive means in the field of evaluation. As the symptoms of a disease comprise a part of the pattern of that disease all these methods present a pattern to show the success or failure (8). The presented methods are adapted from different sciences including psychology, social science, computer and health informatics sciences. Applying adapted evaluation methods requires methodological skills, discipline, innovation and flexibility to adapt the chosen method with the intended case, its situation and specifically information needs (8,24).
The major limitations of this study are that some selected studies deal with only success and failure factors (28,38,42,43,46), or focused on evaluation methods (47), but, due to the significance of these studies, they are considered and reviewed in this research. In addition, among the extracted factors, there were some factors which could be placed in different groups and several discussion sessions took place to choose the most appropriate category for each factor. Another limitation is that only the articles written in English were selected; therefore, there might be some noteworthy articles in this field published in other languages.
In most of the reviewed articles, the main focus has been laid on the necessity of using multi-method approaches and combining methods to obtain more comprehensive and useful results. The integrating of different methods can be beneficial to find an inclusive answer to evaluation questions. The integrity of supplementary methods, data sources, theories and observers are studied under the term of triangulation (58). The combination of qualitative data gathering approaches (such as observations and interviews) and quantitative data gathering approaches (such as questionnaires and work sampling) provides a good opportunity through triangulation to improve the quality of results (59).
Finally, in this research, the extraction of key concepts of each study was carried out through applying meta synthesis, and such concepts were put together and classified, then the suggested framework was formed to evaluate the HIS success [ Table 5 ]. This framework includes 12 main factors, 67 sub factors, and 33 suggested methods for the evaluation of these sub factors. One can determine the appropriate method based on this information and with a general review of the listed methods and factors, or he/she can create better and more comprehensive methods based on his/her information needs. Of course, more attempts and investigations are still necessary to be done in this field.