Evaluation of information systems | Management Optional Notes for UPSC PDF Download

Introduction

• The significance of information technology in various sectors, including business, cannot be overstated, with substantial global investments being made in this field (Seddon, 2001). It is crucial to assess the outcomes of information systems to guide management processes effectively. Evaluation, as defined by Hirschheim and Smithson (1999), is undertaken routinely to measure how well something aligns with specific expectations, objectives, or needs. From a management perspective, Willcocks (1992) describes information system evaluation as the process of determining the value of IT to an organization through quantitative and/or qualitative means. The results of such evaluations inform organizational decisions regarding information system management, particularly investment decisions such as go/no-go choices.
• Evaluations can have various purposes, categorized as either formative or summative (Remenyi, Money, and Sherwood-Smith 2000). Formative evaluations aim to enhance performance and focus on future improvements, while summative evaluations assess past performance quality exclusively. Therefore, evaluations should encompass both monitoring and forecasting aspects. Evaluating information system performance involves assessing hardware, software, computer networks, data, and human resources with the main objective being upgrading and improving maintenance quality. Numerous methodologies for information system evaluation have been proposed in management studies.
• However, information system evaluation is not a simple task; it is a complex process involving multiple dimensions (Peffers and Saarinen, 2002) and various stakeholders (McAulay et al., 2002). Information system investments often yield intangible benefits realized over an extended period (Saarinen and Wijnhoven, 1994). Ad-hoc practices are common in information system evaluation (Irani and Love, 2001), with simple methods like payback period frequently used (Lederer and Mendelow, 1993). Due to its complexity, there are numerous suggestions on how to evaluate IT systems, with much literature adopting a formal-rational view, viewing evaluation primarily as a quantitative process (Walsham, 1993).
• Interpretative approaches view IT systems as social systems with embedded technology (Goldkuhl & Lyytinen, 1982). Peffers and Saarinen (2002) caution that evaluating IT in financial terms may bias toward easily measurable benefits and be prone to manipulation. Contingency models for selecting evaluation methods consider factors such as industry stability, organizational leadership, project types, sizes, expected benefits, and system life cycle stages (Farbey et al., 1992).
• Evaluation of information systems remains a significant topic for study and practice (Irani et al., 2015), especially as IS becomes more pervasive and complex. Evaluating IS success is a critical issue, with ongoing debates on appropriate variables to determine user perceptions (Klecun and Cornford 2015). Peffers and Saarinen (2002) categorize evaluation criteria into five areas: Strategic value, Profitability, Risk, Successful Development and Procurement, and Successful Use and Operations. The IS evaluation process should adapt to changes in assumptions underlying IS investments, with a two-way relationship between the evaluation process, business development, and IS development processes in our conceptual framework.

Types of Strategies

• Goal-Based Evaluation: Goal-based evaluation involves using explicit organizational goals to measure the performance of IT systems. This approach, as described by Patton (1990), focuses on assessing the extent to which a program or intervention achieves specific objectives. Critics, however, argue that this method tends to prioritize technical and economic aspects over human and social aspects, potentially leading to decreased user satisfaction and organizational value (Hirschheim & Smithson, 1988).
• Goal-Free Evaluation: Goal-free evaluation, in contrast, does not rely on explicit goals. Instead, it aims to gain a comprehensive understanding of the subject being evaluated, without predetermined objectives. This approach, described by Patton (1990), emphasizes collecting data on actual effects and evaluating their significance in meeting demonstrated needs. The aim is to avoid bias and maintain objectivity by focusing solely on outcomes and measurable effects.
• Criteria-Based Evaluation: Criteria-based evaluation utilizes explicit general criteria as a yardstick for assessment. Various approaches, such as checklists, heuristics, or quality ideals, are used to evaluate IT systems based on predefined criteria, including the interface, user interaction, and other relevant factors.

Information Systems Evaluation Process

• Recognizing Stakeholder Groups: Before selecting evaluation criteria and methods, it is crucial to identify all relevant stakeholder groups for the Information Systems project (Serafeimidis and Smithson, 2000). This ensures that the evaluation process considers the perspectives of all parties involved.
• Covering Evaluation Criteria: A comprehensive set of evaluation criteria should be used to assess all dimensions of the Information Systems endeavor effectively. These criteria should cover various aspects of the project to ensure a thorough evaluation.
• Incorporation into Business Processes: The Information Systems evaluation process should be integrated into the overall business development, IS development, and procurement processes. This ensures alignment with organizational goals and objectives.

Three-Step Evaluation Process (Wen and Sylla, 1999):

• Intangible Benefits Evaluation: Evaluate intangible benefits and risks associated with IS investments before assessing tangible benefits.
• IS Investment Risk Analysis: Analyze the risks associated with IS investments to inform decision-making.
• Tangible Benefits Evaluation: Evaluate tangible benefits of IS investments after assessing intangible benefits and risks.

Consideration of Organizational Interests: The focus of evaluation may vary depending on organizational interests, such as costs, benefits, competitive position, or industrial relations (Farbey et al., 1992). Evaluators should carefully consider organizational goals and tailor the evaluation process accordingly.

• Communication of Evaluation Results: Evaluation results should be communicated to all stakeholders associated with the project to inform decision-making. This may involve continuing with the investment, making changes to specifications or implementation methods, or discontinuing the project altogether.
• Dimensions of Success Evaluation: Successful IS implementation evaluation should consider both process and product success, including the effectiveness of the development process and the realization of expected benefits (Saarinen, 1993). Evaluation findings can inform learning for future projects and organizational decision-making.

Assessing information systems presents a complex challenge. While it aids organizations in making managerial decisions about their systems, it struggles to adapt to the rapidly evolving landscape of technology.