Enterprise information technologies (EITs), which are strategic systems seeking to integrate the processes and databases of the entire organization and beyond, require a significant investment of money and human resources in return for the promise of a global business model and its associated far-reaching benefits. Their evaluation/justification must be completed with organizational goals and requirements included in the decision, or the organization could lose financially and competitively. Besides traditional financial models, e.g., ROI (return on investment), that are primarily meant for short-term financial justification purposes, there is a paucity of methods for the evaluation of the strategic and intangible costs and benefits that EITs afford organizations as a whole. This article introduces the use of a robust quantitative technique called the analytical hierarchy process (AHP) that can integrate a diverse range of factors (strategic and operational, and tangible and intangible) into one model. The approach can be easily understood by managers and analysts and has a history of application to other types of strategic justification decisions.

ENTERPRISE INFORMATION TECHNOlogies (EITs)(n1) are software systems that change the architecture of information technology (IT) and change and integrate the processes of the entire organization and beyond? The size, scope, and cost of an EIT system are organizationally significant. As a result, the implementation of EIT has primarily been for large corporations that can afford this strategic step, with the hope of gaining long-term advantages. Even with this restricted application, the size of the EIT market, which can be somewhat judged by the revenues of EIT companies themselves, has been massive. For example, the top EIT vendor's current revenue is $3.3 billion.(n3) In addition, EIT vendors are seeking to expand their market base to medium-sized corporations by releasing targeted versions that are cheaper. These indicators have led the market research firm AMR to project the EIT market's revenue to attain $50 billion by the year 2002.(n4)

EITs are strategic technologies whose benefits are distributed throughout the organization and cannot be felt immediately after their implementation. Examples exist where an EIT's short-term costs and the lack of fit with respect to the current systems of an organization have led to the bankruptcy of the organization.(n5) Even in cases when one can see a net improvement attributable to the implementation of EIT, such improvements are often intangible. In fact, according to a recent survey,(n6) while an EIT's long-term annual return can be nearly two-and-one-half times its operating and maintenance costs, it may be the case that the short-term return on an EIT investment is actually negative:

Hence, according to Barry Wilderman, Vice President of the Meta Group, "Traditional measurement methods focused on cost reduction [alone] do not measure the true value of [these] solutions." That is, one also needs to employ techniques for the integrated evaluation of both tangible and intangible factors.(n7)

In a recent article, Holland and Light(n8) make it clear that a successful implementation requires a clear business vision with quantifiable objectives that can be achieved and delivered. Given quantified importance levels of the various factors, this article describes a robust quantitative technique to evaluate EITs on both strategic and operational levels. The importance levels of the factors are obtained through managerial inputs. This process corresponds well with management hierarchy, allowing upper-level management's involvement in the evaluation of strategic factors and line managers evaluating the operational ones. The method can be implemented on a spreadsheet (see Exhibit 1) and is therefore accessible to a wide-ranging audience. For completeness, a framework to manage EITs is presented next.

Since EITs are strategic systems with long-term and broad organizational implications, they need to be evaluated and justified within the context of organizational goals, strategies, and policies. To provide a foundation for the decision evaluation method proposed in the next section, a framework for formulating goals, strategies, and policies is outlined (see Exhibit 2), defining how the EIT-justification step is incorporated within this framework. This formulation process is arduous, but is the necessary effort that must be anticipated for the successful integration of complex and strategic systems into an organization.

In any strategic justification and development framework, the decision for the development and implementation of strategic technologies and programs begins at the upper levels of management. The technology selected should fit well within the vision, goals, and strategic objectives of the organization. To define its goals and strategies, an organization usually undergoes the SWOT-MOSP process, in which Strengths and Weaknesses are assessed in the light of environmental Opportunities and Threats and then Missions, Objectives, Strategies, and Policies are developed.(n9) One of the results of this step in the process is determination of an organization's core competencies.(n10) It is these competencies that need to be supported by the specific technology that is chosen.

At the next level is the initiation of process plans that support the organizational competencies identified earlier and that in turn get supported by the chosen system (EIT or otherwise). Also known as the reengineering phase,(n11) three studies are usually undertaken at this stage: (1) AS-IS, (2) SHOULD-BE, and (3) TO-BE. The AS-IS study provides baseline measures for later justification purposes and provides measures for post-implementation auditing. The SHOULD-BE study exhibits how the current system should function after nonautomation or non-hard technology improvements (simplification and rationalization approaches, such as total quality management, group technology, etc.) are applied to the system. The basic caveat is that a currently disordered system will lead to a disordered EIT system as well. The TO-BE study is used to define the system necessary to meet the objectives set forth by the strategic units.

The next step in the process is to determine what alternate systems configurations (EIT or otherwise) are required. The technical development of various information technology functionalities will need to be addressed at this stage of the process development. The configuration and data relating to alternative systems should finally be integrated at the justification phase of the development cycle.

It is this stage that primarily deals with the analysis to determine the economic, technical, and organizational (operational) feasibility and justification of the system or subsystem. The justification step will have to consider a variety of factors: tangible, intangible, financial, quantitative, and qualitative. Since the analysis of the tangible factors (e.g., finance) is well-studied using methods such as internal rate of return (IRR) and return on investment (ROI), our focus will be on the evaluation of intangible factors based on managerial inputs. To accomplish this task, details on a method known as the analytical hierarchy process (AHP) are described below.

Systems implementation involves:

This last "feedback" stage, although very important from a continuous-improvement perspective, is one of the more neglected steps. Auditing should:

In order to make a decision regarding an EIT implementation, organizations find it necessary to conduct some sort of financial analysis (e.g., ROI) to assess the short-term viability of an EIT investment. However, strategic investments such as EITs are expensive and risky, and they need to fit well within organizational goals, strategies, policies, and systems, as outlined earlier. To evaluate this fit, an array of intangible factors would need to be evaluated and integrated with the more quantitative ones. A well-structured analysis of the intangible factors must involve people with a wide range of experiences and responsibilities (i.e., job functions and management levels). This section provides a decision methodology (AHP) for such an evaluation and integration. AHP, which was developed by Saaty,(n12) has been applied in the past to justify the acquisition of strategic technologies and evaluation of strategic projects.(n13,n14) Yet its application to information technology evaluation is lacking.

There are four major steps in applying the AHP technique:

The AHP model presented here consists of a hierarchy of four levels, illustrated in Exhibit 3. At the top level is the decision or objective itself, and the bottom level consists of the EIT alternatives under evaluation. The factors, which comprise the middle levels, consist of strategic-type factors, which are related to the concerns of upper-level management, as well as EIT-type factors, which are more related to the people who would be managing/using the system.

Strategic Factors for EIT Evaluation. Strategic systems need to be evaluated on strategic performance metrics that are linked to organizational strategy. While financial performance metrics such as stock price and earnings per share can be used to evaluate an organization from a shareholder's perspective, operational strategic-performance metrics should be used to evaluate how an organization is doing with respect to the customer.(n15) It is these strategic performance metrics that are most important from the perspective of strategic project management. These measures include cost, quality, flexibility, and time-based metrics. Varying names have been provided for each of these metrics, but they cover the gamut of organizational and operational performance measures. The relative importance of the performance metrics should be determined relative to overall organizational strategy. Skinner's(n16) "focused" factory paradigm states that certain business characteristics may require a company to put its effort on a subset of the above metrics. For example, a customized medical equipment manufacturer might place a high value on Quality, while in mature industries, Cost may be the most important factor.

EIT Operational Factors. This set of factors relates to the performance of the EIT alternatives at the operational level. The seven factors considered here are as follows.(n17)

The relationship in the hierarchy (represented by the connecting lines in Exhibit 3) between the strategic and operational factors implies that each of the operational factors will have some relative impact on the strategic performance factors. The alternatives of Exhibit 3 and their relationships to the operational factors mean that each of the alternatives will have some level of operational characteristic. These relationships will now need to be evaluated using the next three steps.

The AHP model is usually finalized after a considerable amount of discussion at all levels of management and operations. This discussion may be even more valuable than the actual values assigned because structuring the decision problem helps managers see a logical "whole picture" and the issues involved in such a complex decision. As alluded to earlier, it is possible for two organizations to have a different set of factors for EIT evaluation, and, even if the factors are similar, their importance levels can differ significantly from one organization to another. Given a set of factors impacting the final decision, the purpose of this step is to develop a meaningful manner of extracting user input regarding the decision alternatives. One way to obtain such input is to ask managers to first assign weights to each of the factors and then score the EIT alternatives on these factors. The overall ranking of an EIT alternative i can then be given as follows, where Scoreij is the score of alternative i for factor j and Weightj is the importance of factor j with respect to the decision.

(1) [Multiple line equation(s) cannot be represented in ASCII text]

However, according to cognitive psychological literature(n20), the problem with this approach lies in the difficulty of assigning precise weights on each of the factors, especially when the number of factors or alternatives are in the range of seven or more. Moreover, such a method does not facilitate the structured thinking process necessary for making complex decisions. A more practicable way (which is the AHP-based approach) is to systematically elicit inputs by asking users to evaluate the relative importance of one factor when compared to another factor --pairwise comparisons(n21) -- with respect to a third controlling factor. The controlling factor is one level up in the AHP model. For each controlling factor, the corresponding set of pairwise comparisons (PWCs) can be represented in matrix form. That is, for the relative importance of one strategic factor with respect to other strategic factors, the controlling factor would be the decision itself, and there will be one PWC matrix. For the relative comparisons among EIT Factors, there will be four controlling factors corresponding to the four strategic factors (four PWC matrices), and for the relative comparisons of the EIT Alternatives, there will be seven controlling factors corresponding to the seven EIT Factors (seven PWC matrices). Specifically, for the set of factors involved in Exhibit 3, the characteristics of the PWCs to be made are given in Exhibit 4.

Saaty(n12) suggests that the values assigned to the comparisons of the factors be made in the range 1/9 to 9. A 9 indicates that one factor is extremely more important than the other, a 1/9 indicates that one factor is extremely less important than the other, and a 1 indicates equal importance. Also, if the importance of one factor with respect to a second is given, then the importance of the second factor with respect to the first is the reciprocal; that is, if cost is twice as important as flexibility, then flexibility is half as important as cost. Hence, in column three of Exhibit 4, there is a division by two in the expressions for the number of PWC questions.

From each PWC matrix given in Exhibit 4, we calculate the relative ranking of factors with respect to the corresponding controlling factor. Specifically, for our AHP model, we obtain:

The procedure to obtain the relative importance weights for each PWC matrix is as follows (Saaty(n12) gives more accurate but complex methods):

The overall importance weight of the EIT alternatives (with respect to the decision) is a weighted average of the weights computed in Step 3. That is,

(2) [Multiple line equation(s) cannot be represented in ASCII text]

where weightk,dec is the relative weight of strategic factor k with respect to the decision, weightj,k is the relative weight of EIT Factorj with respect to strategic factor k, and weighti,j is the relative weight of EIT alternative i with respect to EIT factor j.

In order to understand how the calculations work, consider an organization that is evaluating two EIT alternatives. Senior management has decided that cost and quality are the two strategic factors that need to be considered, and middle managers have included the EIT factors: openness and ease of use.(n22) The steps given below correspond to those in the last section. A spreadsheet implementation of the method is given in Exhibit 5.

Step 1. The AHP decision hierarchy model for these incorporating the factors of the illustrative case study is shown in Exhibit 1.

Step 2. The PWCs needed for this example are shown in Exhibit 6. We can see in this example from question 1 that the upper managers believe the organizational drives and goals are more quality focused than cost focused. For the question of how much more important is cost than quality to our strategic decision, we find that cost is much less important at 1/4 or a 0.25 relative weighting. For the reciprocal question of how much more important is quality than cost, a 4 is assigned in the matrix. These values appear in the second column of Exhibit 6. A similar set of PWC matrices is formed for each level of relationships.

Step 3. The calculations for the illustrative example are shown in Exhibit 7. For example, for the first PWC matrix, the column sums obtained from Step 3a are 5 and 1.25, respectively (as given by the shaded elements in column 2). The PWC elements are then divided by the respective column sums as needed by Step 3b (this gives: 1/5 = 0.2; 4/5 = 0.8; 0.25/1.25 = 0.2, and 1/1.25 = 0.8). The relative importance weights of each of the various factors with respect to their controlling factors (Step 3c) are then the average of the elements of the corresponding row, as shown in the fourth column of Exhibit 7. Notice that quality's relative importance is 0.8 and cost's relative importance is 0.2. Rows 4 and 5 of Exhibit 7 show that managers believe Alternative 1 is better than Alternative 2 in terms of openness (0.8 to 0.2 relative weights, respectively) and that Alternative 1 is worse than Alternative 2 in terms of ease of use (0.17 to 0.83).

Step 4. To determine the aggregated values of the operational factors on the decisions (rolling up the weights), first observe from Exhibit 7 that the importance of cost and quality on the decision (Step 4a of Expression 2) is, respectively:

(3) Weightcost, decision = 0.2

Weightquality, decision = 0.8

Using this, the weights of Openness and Ease-of-use on the decision (Step 4b) are given by:

(4) Weightopenness, decision = weightopenness, cost x weightcost, decision weight[sub openness, quality x weightquality, decision

= 0.67 x 0.20 + 0.25 x 0.80 = 0.334

Weightease, decision = weightease, cost x weightcost, decision + weightease, quality x weightquality, decision

= 0.33 x 0.20 + 0.75 x 0.80 = 0.666

The final sets of weights (Step 4c) of the alternatives with respect to the decision are

(5) WeightAlt1, decision = weightAlt1, openness x weightopenness, decision + weightAlt1, ease x weightease, decision

= 0.8 x 0.334 + 0.17 x 0.666 = 0.38

WeightAlt2, decision = weightAlt2, openness x weightopenness, decision + weightAlt2, ease x weightease, decision

= 0.20 x 0.334 + 0.83 x 0.666 = 0.62

This implies that Alternative 2 is the preferred choice for the organization with a 0.62 relative weight.

As with any tool, a number of advantages and disadvantages are associated with this technique. It will benefit the manager and analyst to be aware of some of the major issues. Advantages are:

Disadvantages are

The effort required to manage strategic systems such as EITs can be large. Yet, to ensure long-term viability and appropriate "fit" with the organizational directions, such a process should be envisioned. If organizations are looking to spend millions of dollars for the sake of implementing EIT systems, they will face substantial risk. To minimize this risk, a good strategic framework (as given in this article, for example) is necessary.

Any supporting tool to help evaluate EITs in the context of this framework will have to consider a range of factors: strategic, operational, tangible, and intangible. Exemplary factors and a decision model that integrates them is presented in this article. The modeling approach presented was the analytical hierarchy process (AHP) which has been applied in a number of strategic decision environments. Other models can and should be utilized to help support the fuller context of a true strategic decision. The model was presented in a way that could help managers of information technology to address the issue with relatively little support from a decision analysis expert. Some advantages and disadvantages of the approach were also detailed.

The approach presented here has several implications for management. First, while it is clear that a number of factors must be considered in a strategic decision such as this, it is often unclear as to what the factors must be, what their relationships are, and who may be involved in the decision process. By providing a compendium of factors and integrating them within a decision hierarchy, we have provided a managerial decision-making structure and answer. For this decision environment, strategic factors may be evaluated by senior management, EIT factors may be evaluated by line managers or middle-level managers, and a list of EIT alternatives may be obtained by key end-users.

Second, our approach provides a structured manner of evaluating factors. When a variety of factors are included, the decisionmaking process could quickly get complex and confusing. The approach of considering only two factors at a time (PWC) mitigates this issue. We can also argue that the structured decision environment forcing managers to go through these steps of evaluating the factors and their relationships could be more valuable than the final score or ranking of the alternatives. This process allows managers to understand the full picture in one model, while making sure that incongruencies among levels of management are ironed out.

Third, as illustrated in the case study, the factors presented herein are a guideline, and it is not necessary for an organization to include all them. Keeping in mind the trade-off between a model's complexity and its usefulness, managers must prune the factors that may not be relevant to their particular organizational context, so that any discussion and evaluation of the factors would be fruitful from the decision-making perspective.

PHOTO (BLACK & WHITE): EXHIBIT 1 Spreadsheet Implementation

DIAGRAM: EXHIBIT 2 A Strategic Framework for the Management of EITs

DIAGRAM: EXHIBIT 3 A Detailed Hierarchical Decision Model Incorporating Strategic and Operational Factors

DIAGRAM: EXHIBIT 5 A Simplified Hierarchical Decision Model for the Illustrative Case Study