According to the stages of growth theory, things change over time, in sequential, predictable ways. Eight experts participated in a study that Identified three data warehousing stages of growth -- (1) Initiation, (2) growth, and (3) maturity -- and the variables that define each stage -- data, architecture, stability of the production environment, warehouse staff, users, impact on users' skills and jobs, applications, costs and benefits, and organizational impacts. The data warehousing stages of growth model is useful in understanding and predicting how organizations' data warehouses change. The experts also suggested how recent developments in data warehousing are affecting the stages. Recommendations are given for managing the progression through stages.

TEN YEARS AGO, DATA WAREHOUSING was largely unknown. Today, many companies are receiving considerable business value from their warehousing efforts, as seen in the following examples:

In these and other companies, the benefits from data warehousing did not occur all at once. Typically, there was a specific business problem that motivated the development of the warehouse, such as at Owens & Minor where comprehensive supply chain information was needed; or at First American Corporation where there was a need to integrate data about the bank's customers from a variety of disparate systems; or at Whirlpool where there was the desire to reduce costs, improve quality, and increase customer satisfaction.

Initial success typically leads to an expansion of the warehousing initiative, with more data, applications, and users. Experts say that data warehousing is "a journey, not a destination" in order to emphasize its constantly evolving nature. This ongoing evolution creates many additional organizational opportunities, but also generates issues that must be addressed if the warehouse is to live up to its potential.

Even when a data warehouse reaches maturity, it continues to change. It becomes the foundation for organization-wide reporting systems, predefined and ad hoc queries, decision support systems, executive information systems, and data mining. It becomes critical to performance management (e.g., Balanced Scorecarding), E-commerce (e.g., storing and analyzing clickstream data), and customer relationship management.

In this article, we present a stages of growth model for data warehousing. How a warehouse goes through initiation, growth, and maturity stages and the variables (i.e., characteristics) that define each stage are described. The article also discusses where leading companies are going with their data warehousing efforts. This information should help organizations plan for the evolution of their warehouses. The model and future directions were developed based on interviews with eight leading authorities in the field. These interviews make it clear that while data warehousing has been successful, its greatest impacts are in the future as it becomes integrated with operational processes and E-commerce and is extended to all parties in the value chain. We begin, however, with background information on data warehousing, the stages of growth concept, and how the study was conducted.

During the mid-to-late 1990s, data warehousing became one of the most important developments in the information systems (IS) field. Virtually all of the Fortune 1000 companies now have a data warehouse, and many medium- and small-sized firms are developing them. The Palo Alto Management Group predicts that the data warehousing market will grow to a $113.5 billion market in 2002, including the sales of systems, software, services, and in-house expenditures.(n1) As the new millenium emerged, Year 2000, data warehousing, and electronic commerce were at the top of CIOs' strategic initiatives.(n2)

The rapid growth is due to the combination of business need and technological advances. Businesses are capturing much more data than ever before, especially about their customers, and want to turn this data into actionable information. In the case of FAC, they wanted to know their customers better than anyone else and to use this information to increase profits and value for their customers. This requires collecting, storing, and processing revenue and cost-related data, data about every client transaction, and demographic and psychographic (e.g., client preferences) data about its customers. When data on customer transactions are collected, data warehouses quickly go over a terabyte in size. WalMart's data warehouse is over 125 terabytes in size. Warehouses this large could not exist without recent advances in computer hardware and software technology. Everything must be done in parallel, including the computing hardware and database software.(n3)

A data warehouse (or smaller-scale data mart) is a specially prepared repository of data designed to support decision making. The data comes from operational systems and external sources. To create the data warehouse, data are extracted from source systems, cleaned (e.g., to detect and correct errors), transformed (e.g., put into subject groups or summarized), and loaded into a data store (i.e., placed into a data warehouse). The data in a data warehouse have the following characteristics.(n4)

A data warehouse draws data from operational systems, but is physically separate and serves a different purpose. Operational systems have their own databases and are used for transaction processing; a data warehouse has its own database and is used to support decision making. Once the warehouse is created, users (e.g., analysts, managers) access the data in the warehouse using tools that generate SQL (i.e., structured query language) queries or through applications such as a decision support system or an executive information system. "Data warehousing" is a broader term than "data warehouse" and is used to describe the creation, maintenance, use, and continuous refreshing of the data in the warehouse.

The stages of growth concept is widely used in organizational(n5) and IS research.(n6) The fundamental concept is that many things change over time, in sequential, predictable ways. It has been used to describe, explain, and predict organizational life cycles, product life cycles, and biological growth. In information systems, it has been used with overall computing activities in an organization,(n7) in the evolution of information centers,(n8) and in the integration of information and business systems planning.(n9)

The stages of growth are commonly depicted graphically using an S-shaped curve, where the turnings of the curve mark important transitions. The number of stages varies with the phenomena under investigation, but most models have between three and six stages. Also, over time, additional stages can emerge that were not known or foreseen when the model was first developed. For example, Gibson and Nolan first described a four-stage model(n7) and a couple of years later revised it to include two additional stages.(n10) Each stage is uniquely identified by a set of benchmark variables. These variables change their values as the phenomena move through the stages of evolution.

Data warehouses also evolve over time. For example, the initial rollout of a warehouse is likely to contain data for only a few subject areas and is used by only a subset of the organization's personnel. If the warehouse is successful, additional subject areas and users are added. At the Maturity stage, the warehouse becomes enterprisewide in terms of the data it contains and the user base that it supports.

To develop a stage model for data warehousing, leading experts in the field were contacted and asked to participate in telephone interviews in order to identify the stages and the variables that identify the stages. Every expert who was contacted agreed to participate. The experts included (1) highly regarded and experienced consultants and (2) managers of highly successful data warehouses (see Exhibit 1). Karolyn Duncan has her own consulting firm and teaches courses for The Data Warehousing Institute (TDWI). Jane Griffin heads up Arthur Andersen's data warehousing practice in the Southeast. Bill Inmon is widely regarded as "the father of data warehousing." Theresa Leahy is the data warehousing manager at FAC, which won the 1999 Society for Information Management award for leading practice. Randeen Klarin is the data warehousing manager at NASD Regulations, which won the 1998 TDWI Leadership Award. Don Stoeller received the 1999 TDWI Leadership Award for the data warehouse at Owens & Minor. Ron Swift is a strategist and consultant for NCR CRM Solutions. Jim Thomann is a data warehousing consultant and teaches courses for The Data Warehousing Institute.

The telephone interviews were 20 to 45 minutes in length and focused on three topics:

Based on the information collected, an initial stage model was developed. This model was then sent to the experts for their review and comment. The experts' reactions to the model varied from agreement, to recommendations for minor changes, to suggestions for a model with more stages.(n11) The stage model was revised to reflect the experts' comments.

Three stages describe the current evolution of data warehouses (see Exhibit 2):

There are indicators, however, that there will be additional stages in the future, based on what leading-edge companies are doing with their warehouses. Furthermore, there are iterations within the growth and maturity stages.

Nine variables describe the different stages:

Even though we present a stage model for data warehousing, it is important to recognize that the model is a generalization that does not perfectly describe every company's experiences. Factors such as business need, executive support, and the availability of resources influence how a company's warehousing initiative unfolds. It is also important to recognize that two competing approaches to data warehousing exist. The first is the top-down, enterprise-wide approach advocated by Bill Inmon.(n4) The second is the bottom-up, data mart approach associated with Ralph Kimball.(n12) The latter will be described because it is the more common approach, but the first approach has strong proponents.(n13) Exhibit 3 lists some of the reservations that Bill Inmon has with the data mart strategy. Actions that can be taken to ameliorate the potential negative aspects of the data mart approach will be described. If followed, the actions provide a middle ground between the two competing approaches.

Warehousing initiatives usually start small.(n14) A department, often sales, marketing, or finance, has a need for specific data, and the intended use of the data is clear. In order to meet this need, the building of a data mart is approved (as opposed to a larger data warehouse). When the cost of the mart is relatively modest and it has a strong business sponsor, funding for the project may come from the business unit. The data mart stores only a single or a few subject areas, and the amount of data stored is relatively small. The data are stored in a multidimensional format (e.g., a star schema), so that it matches the way that users think (e.g., customers, products). Given its newness and time pressures to roll out the data mart quickly, populating the mart (via data extraction, transformation, and loading processes) is ad hoc and evolving. To some extent, the data mart is a decision support experiment. The in-house IT personnel assigned to the project have experience with databases, but are typically new to data warehousing. As a result, it is common to bring in consultants to help with the work and provide knowledge transfer. The learning curve for in-house personnel is steep. The initial users of the data mart are analysts in the unit that requested the mart. They are typically more computer literate and savvy than most of the users who will follow. These analysts use a variety of data access tools (e.g., SQL and managed query environments) to access and analyze the data in the mart to address the issues for which the mart was built. The mart allows the analysts to perform their jobs more quickly and thoroughly. Some reporting systems may also be cleaned up and expanded because of the availability of more reliable, consistent, integrated, and timely data. Although the cost of a data mart is less than for a warehouse, it typically runs into at least six figures when hardware, software, and personnel costs are included. The primary benefits provided by the mart are time savings for analysts and IT personnel (e.g., fewer requests for data extracts and special reports), more thorough analyses, answers to specific questions that previously went unanswered, and better decisions. These benefits are realized within the unit that was responsible for building the mart; consequently, the organizational impact is local.

If the data mart proves successful, it provides a "proof of concept" for data warehousing. This typically leads to additional initiatives, such as the expansion of the initial data mart and the demand for additional data marts. The building of more data marts marks the beginning of the growth stage. The company is at a dangerous point. It cannot allow data marts to be developed independently. To do so only perpetuates the "silos of information" problem that is so common in organizations today. There must be an overall plan and architecture for the larger data warehousing initiative. Some of the issues that must be addressed include what are the "official" data sources for the data marts, what data definitions apply across the data marts, and what common dimensions (e.g., product, location, time) are used with the various data marts. The production environment is still somewhat unstable in this stage. Considerable effort is going into expanding the number of data marts and serving a growing user base, so there is little time for formally documenting extraction, transformation, and loading procedures or putting data warehouse performance measurement systems in place. The company's internal data warehousing staff has moved up the learning curve and most of the consultants and vendors are no longer needed. Consultants may still be brought in, but they typically provide highly specialized services, such as database performance tuning. Members of the internal staff assume more specific, specialized roles, such as data modeling new subject areas. As new subject areas and data marts are added, new users come on board. They often are not as savvy as the initial set of users and know less about the data warehousing initiative. They also are more diverse in their information needs and how they want to gain access to the data in the data mart(s) (e.g., custom-built applications, predefined queries). This diversity creates challenges in areas such as end-user training and support and data access tool(s) selection. Some users may find that the requirements of their job (e.g., much more analytical) change so much that they are either unable or unwilling to adapt to the changes and move on to other positions. As people become aware of the potential of data warehousing, new applications emerge. Some of the applications are in the business units served by the new data marts, but there is often a change in the nature of the applications themselves. Whereas the initial applications are often backward facing (i.e., what was), some of the new applications allow users to perform "what-if" analysis about future scenarios. As the number of users and applications grows, so do the benefits, but the benefits still are largely in the form of time savings, new and better information, and improved decision making. At some point in the growth stage (e.g., often about 18 months after the initial rollout), the benefits begin to exceed the costs. Companies often do not know when this occurs because of the time and difficulty of quantifying the benefits. The organizational impact of data warehousing in this stage is still largely tactical rather than strategic. There are exceptions, of course, such as when the warehouse was created to support applications that are an integral part of corporate strategy.

In the maturity stage, the volume of the data maintained grows, it covers multiple subject areas, it is highly detailed, and it provides considerable historical detail. New subject areas are still added, often in three-month iterations. A warehouse is a "journey" not a "destination" in that there is never an end point. To store the data, a data warehouse has replaced most, if not all, of the data marts. There probably still are data marts, but the marts are different from those in previous stages. The earlier marts were "independent" in that they were fed by their own source systems. In the maturity stage, the marts are "dependent" because they are created and fed by an enterprise warehouse. This approach ensures that the organization has a "single version of the truth," because all of the data used for decision support comes from the same source. There are two reasons for creating dependent data marts --(1) improved response time for users and (2) providing users with a simpler view of the data (e.g., just sales data). The data in the warehouse is stored in a relational database (the same kind used with operational systems) in order to manage the vast amount of data efficiently. The multidimensional data model is still used for the data in the dependent data marts. By this time, the production environment is stable. Data extraction, cleansing, transformation, and loading processes are routinized and documented. Good meta data (i.e., data about the data, such as when data in the data warehouse is refreshed) exists. The data warehousing staff is experienced and there is little need for outside consultants. Members of the staff have well-defined roles and responsibilities, including operating and monitoring the warehouse, providing training and support for users, adding new data and applications, and ensuring that the warehouse is aligned with business objectives. The number and range of users continues to grow. Executives may be given an executive information system that draws heavily on warehouse data. Sales reps in the field may be able to access warehouse data over the Web. The data warehousing staff needs to offer courses (e.g., a data warehousing university) and support (e.g., perhaps decentralized to the business units) that is appropriate for the different kinds of users. Almost everyone in the organization needs some minimal competency in accessing and using warehouse data. Many jobs and how they are performed are affected by the warehouse. Applications that use warehouse data can be found throughout the organization. They range from reporting, to predefined and ad hoc queries, to decision support systems, to executive information systems. Data mining applications surface as the organization searches for important relationships (e.g., the characteristics of people who are most likely to respond to a direct mailing) in the massive amount of data stored in the warehouse. Many warehouses become more strategic to the organization as they become important enablers for accomplishing corporate objectives. E-commerce, connecting suppliers and customers along the supply chain, and supporting customer intimacy initiatives are some of the more significant developments that are currently emerging. These applications commonly have warehouse data being used for operational purposes. These recent and important developments are discussed in the next section. The stages and the benchmark variables are summarized in Exhibit 4.

The evolution of data warehousing does not end with the three stages described here. Already, developments are occurring that point to additional stages. Some of the developments are seen in the vignettes that opened this article, while others were suggested by the data warehousing experts. Here are some of the developments that are taking place now or that can be expected in the near future.

Many of the future developments will involve the World Wide Web, and they will occur in a variety of ways. Companies that sell products over the Internet are increasingly collecting, maintaining, and analyzing "clickstream" data -- the mouse clicks at a Web site. This data can be used in two major interrelated ways. The first is with applications that react immediately to the clicks and personalize the shopping experience. What the shopper is presented with is determined "on the fly" based on the shopper's clicks and the intelligence built into the application. The second use is to place clickstream data in a warehouse for analysis purposes. For example, the data might be analyzed to track how people proceed through a Web site, to determine what triggers purchases, to assess what types of information are selected by different categories of shoppers, and to discern what attracts people and keeps them coming back. Data warehousing products are appearing to support these applications.

We can expect data mining to become increasingly common. For years companies knew important relationships (e.g., what products sell together) were hidden in corporate databases, but did not know how to find them. Academicians had methods for discovering the relationships (e.g., neural networks), but there were no software products available to easily apply them. A data warehouse provides the clean, highly granular (i.e., very detailed) data that are required for data mining. Vendors are introducing software that offers "data mining in a box." With this software, the application, such as credit card fraud or market basket analysis, is predetermined. The software provides a set of tools to extract data (often from a data warehouse), analyze the data interactively using an appropriate data-mining algorithm, and present the findings in a way that can be understood. These packaged data mining "solutions" make "rocket science algorithms" usable by companies without "rocket scientists." Many of the data mining applications will be used with customer-centric data warehouses such as at First American Corporation.

To fully understand customers, it is necessary to integrate data from multiple systems and sources. Taking a banking example, data are potentially stored in separate, nonintegrated checking account, credit card, and mortgage loan systems. Additional demographic data about customers are available in external databases. Only by integrating all of this data in a data warehouse is it possible to have a comprehensive understanding of customers. Data warehousing is the critical enabler of one-to-one marketing and customer relationship marketing, and more companies will move their warehouses in this direction.

Another major development will be the closer integration of operational systems and data warehouses. The potential benefits of integration are great. Consider, for example, a call center where a customer representative has access to a profile of every previous interaction between the customer and the company which allows the representative to talk through issues, cross sell, or give additional information. Currently, the information needed for this kind of information is seldom available through a single system.

We have described a stage model for data warehousing that identifies the stages that a data warehouse goes through and the variables that define each stage. We have also considered future directions for data warehousing, directions that suggest a vibrant, important future. By understanding the stage model, managers should be able to better plan their companies' data warehousing initiatives.

When moving through the stages, there are potential pitfalls that should be understood and avoided. In closing, the major ones are discussed.

GRAPH: EXHIBIT 2 The Stages of Growth for Data Warehousing

Even though most firms take the data mart approach, Bill Inmon feels strongly that this is the wrong way to go. According to Inmon, most firms develop four or five marts, only to discover after six to nine months that they need to start over again and develop an enterprise-wide warehouse. The reason for this is that the architecture of marts and warehouses is "genetically" different and marts do not evolve into warehouses. He says, "You don't plant a seed, see it grow into a tumbleweed, and then become an elm." A data warehouse needs to be designed differently from a data mart because of:

These requirements lead to hardware, software, data modeling, and business decisions that are different than those made with marts.