Selecting Project Management Methodologies for ... - Semantic Scholar
Selecting Project Management Methodologies for Business Intelligence Projects – A Value Based Approach Research-in-Progress
Sarah Otyepka University of Augsburg Universitätsstr. 16 86159 Augsburg [email protected]
Benjamin Mosig1 The Boston Consulting Group Kronprinzstraße 28 70173 Stuttgart [email protected]
Marco C. Meier University of Augsburg Universitätsstr. 16 86159 Augsburg [email protected]
Andreas Rathgeber University of Augsburg Alter Postweg 101 86135 Augsburg [email protected]
Abstract While it is common practice to use value based decision models for decisions whether to invest in certain projects or not, there is scarce value based decision support for the selection of the most promising project management methodology to be applied in a specific Business Intelligence project. Addressing the lack of a formal yet practical decision model, this paper proposes a risk-adjusted net-present-value-based model to support decision makers in this particular decision situation. Apart from a project’s estimated cash flows, we focus on two decisive risk parameters – the likelihood of environmental changes and the peril of improper system integration. Using an exemplary calculation, the trade-off between those risks and their impact is formalized and made transparent. Therefore, this paper suggests a decision model to improve the understanding of project management methodology types and hereupon the foundation for the selection of the appropriate one in a specific project setting.
Keywords: Business Intelligence, Value-based Management, Decision Model, Project Management Methodology
1 At the time of writing of a first version of this paper, the author was employed at the FIM Research Center at University of Augsburg.
Project Management and IS Development
Motivation With expected 13 billion USD of spending on Business Intelligence (BI) in 2013 (Gartner, 2012b) and a BI project failure rate of up to 70 % (Adamala and Cidrin 2011; Gartner, 2012a), BI projects represent a remarkable market that still faces challenges. Besides unengaged business sponsors and unavailable or unwilling business representatives, planning activities for BI projects and the development methodology to be applied are among the critical challenges for BI project success (Atre 2003). Therefore, choosing the wrong Project Management Methodology (PMM) in a BI project may result in noticeably higher costs or even project abandonment (Hardy-Vallee 2012; Williams and Williams 2003). Business Intelligence itself is neither a product nor a system, but a ‘broad category of applications, technologies, and processes for gathering, storing, accessing, and analyzing data to help business users make better decisions’ (Watson 2009). Information Systems (IS) can be seen as the technological part of BI, which is complemented with human and organizational skills. Therefore both BI and IS are subcategories of IT (McAfee 2006). Based on the aforementioned definition, we understand “BI projects” as projects aiming at implementing BI. Compared to other stand-alone IT systems BI projects have major differences: They are mostly driven by business opportunity rather than business need and they support cross-organizational decision-support strategy (Moss and Atre 2003). Carroll (2003, citing Avison and Fitzgerald 1995 and Wynekoop and Russo 1997) identifies the following key characteristics of an IS development methodology: ‘it breaks the development process into phases and sub-phases, provides tools, techniques and procedures to assist developers in their work, and has an underlying philosophy that represents a view of the aspects of systems development that are important for ensuring a successful project’. After all, a PMM has to provide techniques to deal with challenges arising during the project, like modifications of requirements that become necessary due to environmental changes (Fowler 2005). Different PMMs offer techniques that focus on dealing with different challenges during projects. We identified that BI projects show important differences compared to other IT or Software Development (SD) projects regarding the degree of planning and flexibility that need to be addressed by an applied PMM. In contrast to code-centric SD projects, BI projects have a data-centric business integration character that demands cross-organizational activities to develop the required enterprise-wide decision support system (Moss and Atre 2003; Moss 2009). This makes a certain degree of planning and coordination activities inevitable (Moss 2009) but at the same time, it is still necessary to provide enough flexibility to react on changes during the project. No PMM provides a silver bullet for project success (Atre 2003; Boehm and Turner 2003; Charvat 2003). A situation-based decision is therefore inevitable to avoid the danger of applying a potentially inappropriate PMM (Charvat 2003). Although the benefit of PMMs is seen in their capability to offer a solution to problematic project delivery, they are infrequently selected on this basis (Wells 2012). We found the decision on PMMs is often rather based on gut feeling or individual preferences than on rational considerations (Charvat 2003) or there is no structured work breakdown at all (Moss and Atre 2003). In other cases companies give mandatory instructions, like authorities that require any in-house IT project to use the V-model XT (Höhn and Höppner 2008). However, the findings of Wells (2012) indicate that a more investigative approach rather than a prescriptive one can enhance the fit of the PMM to the type of the project. Summing this up, we seek an answer to following research question: “How would a value-based decision model look like that supports decision makers in their selection of a type of project management methodology for a given BI project?” Today methods derived from traditional investment theory, such as Return on Investment (ROI), Payback Period, Net Present Value (NPV), Internal Rate of Return (IRR) and Real Options, are widely accepted to quantify the value of IT investments (Benaroch and Kauffman 1999; Verhoef 2005). Therefore it is surprising that the next step, the decision on a PMM to be applied in a specific project setting, is no longer subject to such considerations. Quantitative approaches offer transparency and the opportunity to formally test the model for contradictions. Literature offers scarce quantitative research on this topic whereas numerous specialists or companies publish qualitative best practices. We seek more clarity in the aforementioned decision situation and – addressing the lack of such a decision model –propose to transfer evaluation concepts from investment theory to decisions on the most appropriate PMM in BI projects. These concepts have been developed for evaluating investment alternatives, such as BI projects, and are often requested by executive managers for justifying BI investments (Kimball and Ross 2002).
2
Thirty Fourth International Conference on Information Systems, Milan 2013
Otyepka et al. / Selecting Project Management Methodologies for Business Intelligence Projects
Since suitable and diligent project management can maximize a project’s value to the organization (Charvat 2003) we are thereby able to quantify the value contribution of the different PMMs. Nevertheless we are aware that an evaluation solely based on cost/benefit analysis may disregard other relevant aspects for the decision, which are excluded in favor of a clear analysis of the cost/benefit structure in a first step and might be subject to future research. From a scientific point of view, we expect such a decision model to fulfill the following requirements: It has to (1) be applicable to a class of problems (such as BI projects, in contrast to one specific project), (2) contribute new findings to the body of knowledge, (3) be comprehensible and reproducible, and (4) generate value for a user now or in future (Österle et al. 2011). Furthermore, we additionally expect the model to (5) be economically feasible to ensure its applicability, i.e., the effort of application should be justified by its benefits, and (6) adopt a value-based approach using quantitative metrics as indicated before (Coenenberg et al. 2003; Mertens 1999). This research-in-progress paper is structured as follows. First, it provides necessary theoretical background. The subsequent section introduces the generic value-based decision model and applies the model to a demonstration example. Then, key findings of a first analysis are presented. The last section discusses limitations of the current model and gives an outlook on future research.
Theoretical Background Following a structured literature research strategy, we searched the scientific databases ScienceDirect, EbscoHost, IEEE Xplore, Wiley and AISEL for the terms (Information Systems OR Business Intelligence OR Data Warehouse),(Project OR Management OR Methodology), (Agile OR Plan Driven OR Iterative OR Incremental) and (Selection OR Selecting) including synonyms and combinations of the named tuples. This strategy has been complemented by an unstructured citation searching. Table 1: Summary of approaches for the decision on PMMs Methods from financial investment theory
Application Context
Transferability to a BI- Context
Drawbacks
No
Software Development
Limited, since purely agile approaches are insufficient for BI projects
Addressing agile practices only, no application of investment theory methods
No
Software Development
Transferable; adaption to BI context might be necessary
No value-based approach, no application of investment theory methods
(Cockburn 2000)
No
Software Development
Transferable; adaption to BI context might be necessary
No value-based approach, no application of investment theory methods
(Naumann and Palvia 1982)
No
Decision Support Systems
Transferable; adaption to BI context might be necessary
No application of investment theory methods
Paper
(Mikulenas and Kapocius 2011) (Boehm and Turner 2003)
We extracted papers offering a structured assistance for selecting a PMM (-type) for an IT project. Several papers address the challenge of PMM (-type) selection or application within case studies (Carroll 2003; Mahmood 1987; McHugh and Hogan 2011; Saarinen 1990), offer very generalized characteristics (Charvat 2003) or promote one adjustable roadmap which is supposed to cover any BI project management requirement (Kimball and Ross 2002; Moss and Atre 2003). However, these papers do not offer an
Thirty Fourth International Conference on Information Systems, Milan 2013
3
Project Management and IS Development
artifact like a framework or other specific guidance for the selection of one of several PMMs and are therefore excluded. In terms of guidance for selecting PMMs in IT projects, we identified four relevant papers providing a framework for the selection of PMMs, summarized in Table 1. Although offering a semi-formal approach, combining prioritization techniques with a value-to-cost ratio, Mikulenas and Kapocius (2011) do not offer a fully value-based approach and only consider agile PMMs, which are insufficient to deal with BI projects (Moss and Atre 2003; Moss 2009). Boehm and Turner (2003) in contrast take into account plan-driven, agile and hybrid PMMs and characterize project peculiarities using a qualitative five-dimensional approach to find the most promising PMM. Another qualitative three-dimensional approach is proposed by Cockburn (2000), who suggests a discrete scale for characterizing the project, explicitly not only prior to but also during the project to identify the projects needs regarding the applied PMM. Both last named approaches, though developed for SD projects, can be transferred to BI projects but both of them do not provide the above-claimed value-based character. The evaluation approach of Nauman and Palvia (1982) is an evaluation process including function lists of requirements, scoring techniques and the Delphi Method. Addressing IS projects and not only BI projects and combining qualitative with (not clearly specified) quantitative evaluation, this approach provides a multicriterial selection process but also does not meet the above-mentioned requirements we claim for an evaluation approach. Summarizing our literature research results, we did firstly not find an approach that explicitly addresses BI projects and secondly, there is no quantitative approach based on financial evaluation methods. To use a method deriving from financial investment theory, it is necessary to make a deliberate selection. As indicated in section one, there are numerous publications in the traditional financial investment literature dealing with the selection of whether an investment in general should be carried out or not (Brealey et al. 2011; Copeland et al. 2005). Prerequisite for every value-based decision is defining a scale for measuring the value of every alternative. Quantifying the value of BI in particular comes along with challenges, such as the measurement of intangible values like long term or social effects (Petter et al. 2012). However the assessment of BI value has been subject to several publications and is out of scope for this paper (Adamala and Cidrin 2011; Fitzgerald 1998; Petter et al. 2012; Renkema and Berghout 1997; Walter and Spitta 2004). Mostly, ex ante evaluation and BI project justification in practice tend to rely on financial measurement approaches like cost/benefit analysis (Walter and Spitta 2004). Therefore, we use the same approach to measure the value of the regarded alternatives. We do not consider any options in the project, which excludes the real options approach as additional evaluation method. Efficiency measures like the ROI require reliable estimations of benefits, which is very hard to achieve in a BI context. A Method which is widely accepted for ex ante investment justification is the NPV. Considering the time value of money, the NPV method is always or almost always used by 74% of CFO’s in investment decisions (Graham and Harvey 2001). Therefore, easily applicable yet considering time effects, we prefer the NPV amongst the aforementioned financial evaluation methods. Although widely accepted, the NPV approach so far has not been applied to decisions on the PMM in BI projects.
Decision Model In the following we introduce a basic decision model to determine a project’s expected NPV depending on the applied PMM. Since the number of “brand name” PMMs has been estimated up to 1000 (Jayaratna 1994), the existing multitude of PMMs can hardly be evaluated one by one in this context. Therefore, introducing a classification is a first step to reduce complexity. All popular PMMs applied in IT projects can be assigned to the scale presented in Figure 1. Some PMMs – e. g. eXtreme Programming - may include characteristics of two adjoining types and hence be assigned very close to the boundary. This indicates that defining discrete categories within this continuous scale is challenging. We try to characterize and therefore distinguish four types of PMMS with common patterns: Agile development(A), Incremental development (I), Iterative development (S) and Plan-driven development (P). The currently hyped type referred to as “agile” provides a high degree of flexibility. However, the expression “agile” is used ambiguously in literature. To bring some clarity, we refer to the definition of Conboy (2009): ‘IS development method agility is the continual readiness to rapidly or inherently create change, proactively or reactively embrace change, and learn from change while contributing to perceived customer value (economy, quality, and simplicity), through its collective components and relationships
4
Thirty Fourth International Conference on Information Systems, Milan 2013
Otyepka et al. / Selecting Project Management Methodologies for Business Intelligence Projects
with its environment’. In contrast to A, P promises a high degree of documentation, comparability and planning reliability (Boehm and Turner 2003). Therefore, these two PMMs represent the extremes and illustrate the trade-off between flexibility and planning as indicated in the motivation section. Flexibility
Planning Agile (A)
Incremental (I)
Iterative (S)
Plan-driven (P)
e. g. eXtreme Programming; Crystal; Scrum
e. g. eXtreme Programming; Rational Unified Process (RUP)
e. g. Spiral Model
e. g. Waterfall Method; Cleanroom; SW-CMM; CMMI
Figure 1: Decision tree showing a project’s development structure for every period Iterative and incremental PMMs can be ranked in-between the named extremes, offering a certain degree of control and ex ante planning and the possibility to do rework when releases do not meet the customer’s needs. Although both have in common that they execute all project phases more than once (in contrast to the plan-driven approaches), they must not be used as synonyms. Incremental development I in contrast to S starts with a rough plan of the whole system in the first increment and seeks adding parts of the system in every following increment, comparable to a release-concept (Charvat 2003). This means that parts (releases) of the system are available for the customer early and frequent feedback after every increment is used to develop the system in a goal oriented way, until it fulfills the customer’s requirements. Should the project be abandoned for some reason, the customer can still use the so far developed parts of the system. Therefore the emphasis is on developing the important parts first and then adding to the project or expanding it (Charvat 2003). According to the common NPV approach, one compares the NPVs of two or more alternatives and chooses the alternative generating the highest one – given it is positive (Ross 1995). We assume the decision to run the BI project to be final, therefore the prerequisite of long-term project profitability is fulfilled. A redundant consideration of cash flows that are generated after the end of the project is not necessary since the PMM does not affect the long term profitability of the system in use. Thus, we avoid the risky estimation of mostly intangible or hardly measurable long term benefits and only consider cash flows during project runtime. Consequently, the resulting NPV is expected to be negative in our demonstration example. Validation is hard to achieve, since there is no empirical data including the same project run with different project management methodologies that could provide insights for future decision situations. At the same time it is not possible to provide identical pre-conditions to conduct an experiment for empirical purposes. Facing a lack of data history, we follow a design-oriented approach for developing the decision model. To validate the model we use a demonstration example with project cash flows approved by BI development experts in a first step. Hereby, we seek a better understanding of project management methodologies in general and their application in BI projects. In this research-in-progress paper, we take into account two risk parameters. The first refers to the abovementioned trade-off of planning and flexibility. Any project faces environmental changes or the need to refine customer requirements, which the applied PMM has to deal with. This is the major shortcoming associated with P, since it is widely criticized for being inappropriate to handle rapidly changing environments and customer requirements (Abrahamsson et al. 2009; Beck 1999; Charvat 2003; McCracken and Jackson 1982). I and S are more in the position to react on changes due to their release structure (Beck 1999) and A emphasizes close customer collaboration and is therefore even supposed to “embrace change” (Beck 1999; Boehm and Turner 2003; Conboy 2009; Fowler and Highsmith 2001). A, in contrast, is characterized as lightweight processes with close customer collaboration and short iterative cycles resulting in independently running modules (Boehm and Turner 2003; Fowler and
Thirty Fourth International Conference on Information Systems, Milan 2013
5
Project Management and IS Development
Highsmith 2001). The trend to value working software over documentation, individuals and interactions over processes and tools and responding to change over following a plan bears the risk of delivering perfectly working individual modules which are not thoroughly working as an integrated system (Black et al. 2009). If applying I or S, one can handle this requirement better since these PMMs include a higher level of planning activities, but following the prototyping approach, this risk is still considerable. Since P includes a high level of cross-organizational planning activities, this PMM bears the lowest risk of a lack of integration. Therefore, environmental dynamism δ and improper system integration εk are the two major risks to be taken into account in the proposed decision model. Environmental dynamism describes changes in a defined period of time within the project environment which require changes of the project’s management. Improper system integration results when parts of the system do not fit together at the end of the project and rework has to be done to fix the problem. Mostly, risk is perceived as negative or harmful deviation from the expected status. Since this is the most likely situation in BI projects, we only consider negative development in our decision model. Even if positive development is a possible scenario, this case is not necessary for the overall understanding of the model and might be subject to continuative research projects. The simplified decision tree presented in Figure 2 shows the possible development of a BI project. Like indicated before, we do not consider enhancing factors in a first step. In case a risk occurs, the decision of the company is to fix the failure (see A5).
t0
L= 1- δ
L= δ
t1
…
t1
…
Risk of environmental changes δ in every Period tn, n ϵ {1, …, T}
tn
L= 1- εk
L= εk
T
Risk-Neutral NPV See (Ia) below
T
Disturbing Factors See (Ib), (Ic) below
Risk of a lack of integration εk in the last Period T
Figure 2: Decision tree showing a project’s development structure for every period The underlying model needs more detailed explanation and is based on following formal assumptions: (A1) The decision to implement a BI system is final. The company only chooses between a preselected number of PMMs k K {A, I, S, P}. We assume that any chosen PMM is applied accurately. Due to its distinct strengths and weaknesses, each PMM has a different cost/benefit structure that is captured in aggregated cash flows per period. (A2) The model does not take into account the possibility of project abandonment in form of real options or the possibility of switching to another PMM during the project and can therefore be seen as a snap-shot of the currently expected situation. It is possible that repeated calculations at any point later during the project indicate that one of the (other) named actions might be beneficial. (A3) The project is characterized by two types of risks. These are the likelihood of environmental changes δ [0;1] within each project period and the likelihood of a lack of system integration εk [0;1] at the end of the project. We assume both risks to be independent. (A4) Since the periods are of equal length, δ can be assumed to be constant in every period and independent of the chosen PMM. εk is inherent to the chosen PMM and hence differs, denoted by index k. δ occurs in each period, εk only occurs in the last period T. (A5) If a risk occurs, the company will fix the failure. The costs associated with the correction are ascribed to the period the risk occurred. The costs to react on Dynamism, CDyn,k are involved
6
Thirty Fourth International Conference on Information Systems, Milan 2013
Otyepka et al. / Selecting Project Management Methodologies for Business Intelligence Projects
with δ and their amount depends on the regarded PMM. The cost for necessary rework due to a lack of system integration, CInt, are involved with εk. CInt can be assumed to be equal for any PMM.
Figure 3 can be transferred to the general model presented in Formula I:
NPVk ,risk adj NPVk ,exp ected *
C Int (1 i ) T 1 * C Dyn,k k * T (1 i ) * i (1 i ) T
(Ia)
(Ib)
(I)
(Ic)
Equation (Ia) denotes the common risk-free NPV calculated according to the project’s estimated cash flows. Equation (Ib) denotes the expected NPV of additional costs due to environmental changes which occur with the likelihood δ. Equation (Ic) denotes the expected NPV of additional costs coming along with the risk of improper system integration which we assume only to occur in the last period. (Ib) and (Ic) therefore represent disturbing factors which can downgrade the project’s NPV. To get the total expected NPVk,risk-adj, we calculate the sum of equations (Ia) to (Ic). Formula (I) allows us to evaluate a BI project depending on the applied PMM. In the next section we introduce a demonstration example to illustrate the model’s application and its benefits.
Demonstration Example For a better understanding of the proposed decision model, we apply it to a demonstration example according to Formula (I). The decision to run the BI-project is final, the estimation for project duration is 12 months with total costs of 2.000.000,00 USD. Seeking recommendation on the overall management approach, the preselection includes four types of development approaches namely A, I, S and P. Although practitioners have found P to be insufficient for BI-projects (Moss and Atre 2003; Moss 2009), we include it in our analysis for the sake of completeness. We do not narrow the selection down to particular methodologies like SCRUM, Capability Maturity Model (CMM) or others in this paper. This subsequent selection of a particular methodology might include qualitative parameters like company culture and project criticality (Boehm and Turner 2003), that are often included in non-value-based decision models. This is too far reaching to be addressed in this research-in-progress paper and might be matter of future research. Depending on the applied PMM, we split the estimated total costs of the project to the periods as shown in Figure 3.We assume total costs of 2,000,000.00 USD, split to the 12 periods and affecting net income in the period they occur. We assume the costs to be distributed according to the findings of Black et al. (2009). This means in case of applying A, we assume an iteration length of one month and equally high costs of $ 170.000,00 per period. In case of applying I, we assume a duration of each increment of 2 months and higher costs at the end of each increment. In case of applying S, we assume an iteration length of 3 months. Due to the fact that in iteration applies a short version of the plan driven development approach (Moss 2009), we assume slightly decreasing costs within every iteration (Black et al. 2009). Accordingly, we assume slightly decreasing costs across project duration in case of applying P. The interest rate of 1% per month, resulting in an annual interest rate of 12,7% is notably higher than the money market interest and makes allowance for risk averse stakeholders. The expected additional costs due to the risk structure approximate 20% of the estimated costs, a rate which is generally assumed by consultancies and varies slightly depending on the applied PMM.
Thirty Fourth International Conference on Information Systems, Milan 2013
7
Project Management and IS Development
t
Estimated Costs Agile
Increm ental
1
$1 7 0.000,00
2
$1 7 0.000,00
Spiral
Risk-Dependent Costs Agile
Plan-driv en
$1 40.000,00 $220.000,00 $230.000,00 Cdy n , per month $200.000,00 $1 7 0.000,00 $220.000,00 CInt , one-time $1 40.000,00 $1 1 0.000,00 $220.000,00 Likelihood for $200.000,00 $220.000,00 $21 0.000,00 $1 40.000,00 $1 7 0.000,00 $1 80.000,00 a lack of sy stem integration $200.000,00 $1 1 0.000,00 $1 7 0.000,00
$1 5.000,00
Increm ental
Spiral
Plan-driv en
$24.000,00 $30.000,00
$40.000,00
$400.000,00
3
$1 7 0.000,00
4
$1 7 0.000,00
5
$1 7 0.000,00
6
$1 7 0.000,00
7
$1 7 0.000,00
$1 40.000,00 $220.000,00 $1 7 0.000,00
T=
12
8
$1 7 0.000,00
$200.000,00 $1 7 0.000,00 $1 60.000,00
i=
0,01 per month
9
$1 7 0.000,00
$1 40.000,00 $1 1 0.000,00 $1 20.000,00
10
$1 7 0.000,00
$200.000,00 $220.000,00
$1 1 0.000,00
11
$1 7 0.000,00
$1 40.000,00 $1 7 0.000,00
$1 1 0.000,00
12
$1 30.000,00
$1 60.000,00 $1 1 0.000,00 $1 00.000,00
ε+0,1
ε+0,05
ε+0,025
ε
Figure 3: Not-discounted costs split to the project’s periods Figure 4 shows the calculation results for all parameter combinations of δ and εk in 0.1 steps and gives a recommendation for the most appropriate PMM type. δ: Likelihood of environmental changes 0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
0 0,1 0,2 0,3
ε: Likelihood of a lack of system integration
0,4 0,5 0,6 0,7
Incremen‐ Iterative tal Develop‐ Develop‐ ment (S) ment (I)
Agile Development (A)
0,8 0,9 1
(I)
Figure 4: Exemplary matrix showing calculation results The advantage of this approach is that risk parameters do not have to be determined ex ante. Using this matrix, a decision maker gets an idea of the overall situation and can decide according to the provided information. It shows that practitioners’ estimation on plan driven development in BI-projects is confirmed, since there is no parameter-combination that shows P to be most appropriate. A is preferred for very high likelihood of environmental changes. S and I is more appropriate in moderately changing environments. Although a purely agile approach has been claimed to be inappropriate for BI projects as well (Moss 2009), the calculation shows that an agile approach is most promising for the majority of parameter combinations. This might be an indicator that the ability to embrace change is of high importance for BI projects. The result also indicates that the risk of dynamism outweighs the risk of system integration. One reason for this observation might be the fact that a flexible and highly agile project environment enables project leaders to fix integration issues once they are identified.
Practical Implications, Limitations and Conclusion so far The presented model fulfils all six initially stated requirements mentioned in section 2. (1) It is applicable to a class of problems, namely to decisions on the PMM type in a BI project. (2) It contributes to the body of knowledge since no formal quantitative model on the decision between agile and plan-driven PMMs could be found in literature. (3) The underlying NPV approach is established and results are easily reproducible. (4) The model adds value to the decision process by raising the clarity in decision situations and therefore improving the decision itself. (5) Since estimating the project’s cash flows is necessary for project planning anyway, the model is easily applicable. Decision makers have to only estimate two additional risk parameters. (6) Relying on a NPV-based decision model, we propose to complement
8
Thirty Fourth International Conference on Information Systems, Milan 2013
Otyepka et al. / Selecting Project Management Methodologies for Business Intelligence Projects
decision strategies on a PMM in BI projects by a quantitative, value-based and future-oriented approach. Nevertheless, the proposed model is beset with limitations that need to be taken into account when applying it in project settings: The proposed model does not provide decision support on a specific PMM and is only able to distinguish types of PMMs. This classification however is a necessary first step which makes it possible to analyse the associated PMMs for the final decision. Furthermore, the proposed model does not take into account project risks which frequently threaten project success, such as unavailable or unwilling business representatives or unengaged business sponsors and a lack of understanding for the project situation. Also, deficient methodology application and usage is not considered in the presented model although this is an important reason for project failure. Implementation risks like determining a project’s goals and requirements come along with any PMM and project. Since this is not a factor for differentiating the PMMs in our decision model, we cannot consider them. These risks are out of scope of our work and have to be addressed separately. Moreover, the proposed model only takes into account two parameters which might neglect important intangible values and social effects. This limitation might be addressed in the future by integrating more parameters, for example the parameters proposed by Boehm and Turner (2003). Finally, the proposed model claims to offer a clear recommendation for a decision, yet is prone to errors of the estimated underlying cash flows. To strengthen the reliability of the decision model, we plan to address this shortcoming by conducting further sensitivity analyses to test the impact of estimation errors regarding both, cash flows and risk parameters. We further plan to validate the model in collaboration with BI consultants by applying the model in the beginning of BI projects parallel to the consultancy’s established approach. Comparing the results of the proposed decision model with the decisions made during the project might offer interesting results. The idea of using the presented model in order to uncover formal contradictions to bring more clarity and understanding to the decision problem on PMMs in BI projects might be helpful in other IT projects as well. Doing so, one still has to consider that the parameters used for BI projects might not be applicable to another kind of project (like SD projects). It is necessary to verify if the model parameters are applicable to other projects and to adjust them if needed. Despite these limitations, the proposed formal NPV model is one way to bring more transparency to the described decision situation. Although not all kind of project risks are addressed, first trying to understand the project environment and then applying a diligent selection of a PMM should help to avoid project failure that goes back to the application of an inappropriate PMM.
References Abrahamsson, P., Conboy, K., and Wang, X. 2009. "'Lots Done, More To Do': the Current State of Agile Systems Development Research," European Journal of Information Systems (18:4), pp. 281-284. Adamala, S., and Cidrin, L. 2011. "Key Success Factors in Business Intelligence," Journal of Intelligence Studies in Business (1:1), pp. 107-127. Atre, S. 2003. "The Top 10 Critical Challenges for Business Intelligence Success," http://www.computerworld.com/computerworld/records/images/pdf/BusIntellWPonline.pdf. Beck, K. 1999. "Embracing Change with Extreme Programming," IEEE Computer (10:32), pp. 70-77. Benaroch, M., and Kauffman, R. 1999. "A Case for Using Real Options Pricing Analysis to Evaluate Information Technology Project Investments," Information Systems Research (1:10), pp. 70-86. Black, S., Boca, P. P., Bowen, J. P., Gorman, J., and Hinchey, M. 2009. "Formal versus Agile: Survival of the Fittest," IEEE Computer (9:42), pp. 37-45. Boehm, B. W., and Turner, R. 2003. Balancing Agility and Discipline: A Guide for the Perplexed, Addison-Wesley Longman Publishing Co., Inc. Brealey, R. A., Myers, S. C., and Allen, F. 2011. Principles of Corporate Finance, McGraw-Hill/Irwin.
Thirty Fourth International Conference on Information Systems, Milan 2013
9
Project Management and IS Development
Carroll, J. 2003. "The Process of ISD Methodology Selection and Use: A Case Study." Proceedings of the 11th European Conference on Information Systems, ECIS , Naples, Italy, pp. 379-392. Charvat, J. 2003. Project Management Methodologies: Selecting, Implementing, and Supporting Methodologies and Processes for Projects, Hoboken, NJ: Wiley. Cockburn, A. 2000. "Selecting a Project's Methodology," IEEE Software (4:17), pp. 64-71. Coenenberg, A. G., Mattner, G. R., and Schultze, W. 2003. "Wertorientierte Steuerung: Anforderungen, Konzepte, Anwendungsprobleme," in Finanzwirtschaft, Kapitalmarkt und Banken, Andreas Rathgeber, H. Tebroke and M. Wallmeier (eds.) Stuttgart, Germany, Schäffer-Poeschel, pp. 1-24. Conboy, K. 2009. "Agility from First Principles: Reconstructing the Concept of Agility in Information Systems Development," Information Systems Research (3:20), pp. 329-354. Copeland, T. E., Weston, J. F., and Shastri, K. 2005. Financial Theory And Corporate Policy, London, UK, Pearson. Fitzgerald, G. 1998. "Evaluating Information Systems Projects: A Multidimensional Approach," Journal of Information Technology (1:13), pp. 15-27. Fowler, M., and Highsmith, J. 2001. "The Agile Manifesto," Software Development (8:9), pp. 28-35. Fowler, M. 2005. "The New Methodology," http://www.martinfowler.com/articles/newMethodology.html. Gartner, Inc., Stamford, Connecticut 2012a. "Fewer than 30 Percent of Business Intelligence Initiatives will Align Analytic Metrics Completely with Enterprise Business Drivers by 2014," http://www.gartner.com/it/page.jsp?id=1891515.Gartner Research Gartner, Inc., Stamford, Connecticut 2012b. "Worldwide Spending on Enterprise Application Software to Increase 4.5 Percent in 2012 ," http://www.gartner.com/it/page.jsp?id=2055215. Gartner Research Graham, J. R., and Harvey, C. R. 2001. "The Theory and Practice of Corporate Finance: Evidence from the Field," Journal of Financial Economics (60), pp. 187-243. Hardy-Vallee, B. 2012. "The Cost of Bad Project Management," http://businessjournal.gallup.com/content/152429/cost-bad-project-management.aspx#2. Höhn, R., and Höppner, S. 2008. Das V-Modell XT: Grundlagen, Methodik und Anwendungen, Heidelberg,Germany, Springer. Jayaratna, N. 1994. Understanding and Evaluating Methodologies: NIMSAD, a Systematic Framework, New York City, USA, McGraw-Hill, Inc. Kimball, R., and Ross, M. 2002. The Data Warehouse Toolkit: The Complete Guide to Dimensional Modeling 2002, Hoboken, NJ: Wiley. Mahmood, M. A. 1987. "System Development Methods—a Comparative Investigation," MIS Quarterly (3:11), pp. 293-311. McAfee, A. 2006. "Mastering the Three Worlds of Information Technology," Harvard Business Review. (11:84), pp. 141. McCracken, D. D., and Jackson, M. A. 1982. "Life Cycle Concept Considered Harmful," ACM SIGSOFT Software Engineering Notes (2:7), pp. 29-32.
10 Thirty Fourth International Conference on Information Systems, Milan 2013
Otyepka et al. / Selecting Project Management Methodologies for Business Intelligence Projects
McHugh, O., and Hogan, M. 2011. "Investigating the Rationale for Adopting an InternationallyRecognised Project Management Methodology in Ireland: The view of the Project Manager," International Journal of Project Management. (5:29), pp. 637-646. Mertens, P. 1999. "Operiert die Wirtschaftsinformatik mit den falschen Unternehmenszielen? - 15 Thesen," in Wirtschaftsinformatik und Wissenschaftstheorie, Jörg Becker, W. König, R. Schütte, O. Wendt and S. Zelewski (eds.) Wiesbaden, Germany, Gabler, pp. 379-392. Mikulenas, G., and Kapocius, K. 2011. "An Approach for Prioritizing Agile Practices for Adaptation," Information Systems Development, pp. 485-498. Moss, L. T., and Atre, S. 2003. Business Intelligence Roadmap: The Complete Project Lifecycle for Decision-Support Applications, Boston, USA, Addison-Wesley Professional. Moss, L. T. 2009. "Beware of Scrum Fanatics On DW/BI Projects,"Enterprise Information Management Insight MAGAZINE, (3:3). Naumann, J. D., and Palvia, S. 1982. "A selection Model for Systems Development Tools," MIS Quarterly pp. 39-48. Österle, H., Becker, J., Frank, U., Hess, T., Karagiannis, D., Krcmar, H., Loos, P., Mertens, P., Oberweis, A., and Sinz, E. 2011. "Memorandum on Design-Oriented Information Systems Research," European Journal of Information Systems (1:20), pp. 7-10. Petter, S., DeLone, W., and McLean, E. R. 2012. "The Past, Present, and Future of “IS Success”," Journal of the Association for Information Systems (5:13), pp. 341-362. Renkema, T. J. W., and Berghout, E. W. 1997. "Methodologies for Information Systems Investment Evaluation at the Proposal Stage: A Comparative Review," Information and Software Technology (1:39), pp. 1-13. Ross, S. A. 1995. "Uses, Abuses, and Alternatives to the Net-Present-Value Rule," Financial Management (24:3), pp. 96-102. Saarinen, T. 1990. "System Development Methodology and Project Success: An Assessment of Situational Approaches," Information & Management (3:19), pp. 183-193. Verhoef, C. 2005. "Quantifying the Value of IT-Investments," Science of Computer Programming (3:56), pp. 315-342. Walter, S. G., and Spitta, T. 2004. "Approaches to the Ex-Ante Evaluation of Investments into Information Systems," WIRTSCHAFTSINFORMATIK (3:46), pp. 171-180. Watson, H. J. 2009. "Tutorial Business Intelligence - Past, Presence, and Future," Communications of the AIS (1:25), pp. 487-510. Wells, H. 2012. "How Effective Are Project Management Methodologies? An Explorative Evaluation of Their Benefits in Practice," Project Management Jounal (6:43), pp. 43-58. Williams, S., and Williams, N. 2003. "The Business Value of Business Intelligence," Journal of Data Warehousing (4:8), pp. 30-39.
Thirty Fourth International Conference on Information Systems, Milan 2013
11
Sarah Otyepka University of Augsburg Universitätsstr. 16 86159 Augsburg [email protected]
Benjamin Mosig1 The Boston Consulting Group Kronprinzstraße 28 70173 Stuttgart [email protected]
Marco C. Meier University of Augsburg Universitätsstr. 16 86159 Augsburg [email protected]
Andreas Rathgeber University of Augsburg Alter Postweg 101 86135 Augsburg [email protected]
Abstract While it is common practice to use value based decision models for decisions whether to invest in certain projects or not, there is scarce value based decision support for the selection of the most promising project management methodology to be applied in a specific Business Intelligence project. Addressing the lack of a formal yet practical decision model, this paper proposes a risk-adjusted net-present-value-based model to support decision makers in this particular decision situation. Apart from a project’s estimated cash flows, we focus on two decisive risk parameters – the likelihood of environmental changes and the peril of improper system integration. Using an exemplary calculation, the trade-off between those risks and their impact is formalized and made transparent. Therefore, this paper suggests a decision model to improve the understanding of project management methodology types and hereupon the foundation for the selection of the appropriate one in a specific project setting.
Keywords: Business Intelligence, Value-based Management, Decision Model, Project Management Methodology
1 At the time of writing of a first version of this paper, the author was employed at the FIM Research Center at University of Augsburg.
Project Management and IS Development
Motivation With expected 13 billion USD of spending on Business Intelligence (BI) in 2013 (Gartner, 2012b) and a BI project failure rate of up to 70 % (Adamala and Cidrin 2011; Gartner, 2012a), BI projects represent a remarkable market that still faces challenges. Besides unengaged business sponsors and unavailable or unwilling business representatives, planning activities for BI projects and the development methodology to be applied are among the critical challenges for BI project success (Atre 2003). Therefore, choosing the wrong Project Management Methodology (PMM) in a BI project may result in noticeably higher costs or even project abandonment (Hardy-Vallee 2012; Williams and Williams 2003). Business Intelligence itself is neither a product nor a system, but a ‘broad category of applications, technologies, and processes for gathering, storing, accessing, and analyzing data to help business users make better decisions’ (Watson 2009). Information Systems (IS) can be seen as the technological part of BI, which is complemented with human and organizational skills. Therefore both BI and IS are subcategories of IT (McAfee 2006). Based on the aforementioned definition, we understand “BI projects” as projects aiming at implementing BI. Compared to other stand-alone IT systems BI projects have major differences: They are mostly driven by business opportunity rather than business need and they support cross-organizational decision-support strategy (Moss and Atre 2003). Carroll (2003, citing Avison and Fitzgerald 1995 and Wynekoop and Russo 1997) identifies the following key characteristics of an IS development methodology: ‘it breaks the development process into phases and sub-phases, provides tools, techniques and procedures to assist developers in their work, and has an underlying philosophy that represents a view of the aspects of systems development that are important for ensuring a successful project’. After all, a PMM has to provide techniques to deal with challenges arising during the project, like modifications of requirements that become necessary due to environmental changes (Fowler 2005). Different PMMs offer techniques that focus on dealing with different challenges during projects. We identified that BI projects show important differences compared to other IT or Software Development (SD) projects regarding the degree of planning and flexibility that need to be addressed by an applied PMM. In contrast to code-centric SD projects, BI projects have a data-centric business integration character that demands cross-organizational activities to develop the required enterprise-wide decision support system (Moss and Atre 2003; Moss 2009). This makes a certain degree of planning and coordination activities inevitable (Moss 2009) but at the same time, it is still necessary to provide enough flexibility to react on changes during the project. No PMM provides a silver bullet for project success (Atre 2003; Boehm and Turner 2003; Charvat 2003). A situation-based decision is therefore inevitable to avoid the danger of applying a potentially inappropriate PMM (Charvat 2003). Although the benefit of PMMs is seen in their capability to offer a solution to problematic project delivery, they are infrequently selected on this basis (Wells 2012). We found the decision on PMMs is often rather based on gut feeling or individual preferences than on rational considerations (Charvat 2003) or there is no structured work breakdown at all (Moss and Atre 2003). In other cases companies give mandatory instructions, like authorities that require any in-house IT project to use the V-model XT (Höhn and Höppner 2008). However, the findings of Wells (2012) indicate that a more investigative approach rather than a prescriptive one can enhance the fit of the PMM to the type of the project. Summing this up, we seek an answer to following research question: “How would a value-based decision model look like that supports decision makers in their selection of a type of project management methodology for a given BI project?” Today methods derived from traditional investment theory, such as Return on Investment (ROI), Payback Period, Net Present Value (NPV), Internal Rate of Return (IRR) and Real Options, are widely accepted to quantify the value of IT investments (Benaroch and Kauffman 1999; Verhoef 2005). Therefore it is surprising that the next step, the decision on a PMM to be applied in a specific project setting, is no longer subject to such considerations. Quantitative approaches offer transparency and the opportunity to formally test the model for contradictions. Literature offers scarce quantitative research on this topic whereas numerous specialists or companies publish qualitative best practices. We seek more clarity in the aforementioned decision situation and – addressing the lack of such a decision model –propose to transfer evaluation concepts from investment theory to decisions on the most appropriate PMM in BI projects. These concepts have been developed for evaluating investment alternatives, such as BI projects, and are often requested by executive managers for justifying BI investments (Kimball and Ross 2002).
2
Thirty Fourth International Conference on Information Systems, Milan 2013
Otyepka et al. / Selecting Project Management Methodologies for Business Intelligence Projects
Since suitable and diligent project management can maximize a project’s value to the organization (Charvat 2003) we are thereby able to quantify the value contribution of the different PMMs. Nevertheless we are aware that an evaluation solely based on cost/benefit analysis may disregard other relevant aspects for the decision, which are excluded in favor of a clear analysis of the cost/benefit structure in a first step and might be subject to future research. From a scientific point of view, we expect such a decision model to fulfill the following requirements: It has to (1) be applicable to a class of problems (such as BI projects, in contrast to one specific project), (2) contribute new findings to the body of knowledge, (3) be comprehensible and reproducible, and (4) generate value for a user now or in future (Österle et al. 2011). Furthermore, we additionally expect the model to (5) be economically feasible to ensure its applicability, i.e., the effort of application should be justified by its benefits, and (6) adopt a value-based approach using quantitative metrics as indicated before (Coenenberg et al. 2003; Mertens 1999). This research-in-progress paper is structured as follows. First, it provides necessary theoretical background. The subsequent section introduces the generic value-based decision model and applies the model to a demonstration example. Then, key findings of a first analysis are presented. The last section discusses limitations of the current model and gives an outlook on future research.
Theoretical Background Following a structured literature research strategy, we searched the scientific databases ScienceDirect, EbscoHost, IEEE Xplore, Wiley and AISEL for the terms (Information Systems OR Business Intelligence OR Data Warehouse),(Project OR Management OR Methodology), (Agile OR Plan Driven OR Iterative OR Incremental) and (Selection OR Selecting) including synonyms and combinations of the named tuples. This strategy has been complemented by an unstructured citation searching. Table 1: Summary of approaches for the decision on PMMs Methods from financial investment theory
Application Context
Transferability to a BI- Context
Drawbacks
No
Software Development
Limited, since purely agile approaches are insufficient for BI projects
Addressing agile practices only, no application of investment theory methods
No
Software Development
Transferable; adaption to BI context might be necessary
No value-based approach, no application of investment theory methods
(Cockburn 2000)
No
Software Development
Transferable; adaption to BI context might be necessary
No value-based approach, no application of investment theory methods
(Naumann and Palvia 1982)
No
Decision Support Systems
Transferable; adaption to BI context might be necessary
No application of investment theory methods
Paper
(Mikulenas and Kapocius 2011) (Boehm and Turner 2003)
We extracted papers offering a structured assistance for selecting a PMM (-type) for an IT project. Several papers address the challenge of PMM (-type) selection or application within case studies (Carroll 2003; Mahmood 1987; McHugh and Hogan 2011; Saarinen 1990), offer very generalized characteristics (Charvat 2003) or promote one adjustable roadmap which is supposed to cover any BI project management requirement (Kimball and Ross 2002; Moss and Atre 2003). However, these papers do not offer an
Thirty Fourth International Conference on Information Systems, Milan 2013
3
Project Management and IS Development
artifact like a framework or other specific guidance for the selection of one of several PMMs and are therefore excluded. In terms of guidance for selecting PMMs in IT projects, we identified four relevant papers providing a framework for the selection of PMMs, summarized in Table 1. Although offering a semi-formal approach, combining prioritization techniques with a value-to-cost ratio, Mikulenas and Kapocius (2011) do not offer a fully value-based approach and only consider agile PMMs, which are insufficient to deal with BI projects (Moss and Atre 2003; Moss 2009). Boehm and Turner (2003) in contrast take into account plan-driven, agile and hybrid PMMs and characterize project peculiarities using a qualitative five-dimensional approach to find the most promising PMM. Another qualitative three-dimensional approach is proposed by Cockburn (2000), who suggests a discrete scale for characterizing the project, explicitly not only prior to but also during the project to identify the projects needs regarding the applied PMM. Both last named approaches, though developed for SD projects, can be transferred to BI projects but both of them do not provide the above-claimed value-based character. The evaluation approach of Nauman and Palvia (1982) is an evaluation process including function lists of requirements, scoring techniques and the Delphi Method. Addressing IS projects and not only BI projects and combining qualitative with (not clearly specified) quantitative evaluation, this approach provides a multicriterial selection process but also does not meet the above-mentioned requirements we claim for an evaluation approach. Summarizing our literature research results, we did firstly not find an approach that explicitly addresses BI projects and secondly, there is no quantitative approach based on financial evaluation methods. To use a method deriving from financial investment theory, it is necessary to make a deliberate selection. As indicated in section one, there are numerous publications in the traditional financial investment literature dealing with the selection of whether an investment in general should be carried out or not (Brealey et al. 2011; Copeland et al. 2005). Prerequisite for every value-based decision is defining a scale for measuring the value of every alternative. Quantifying the value of BI in particular comes along with challenges, such as the measurement of intangible values like long term or social effects (Petter et al. 2012). However the assessment of BI value has been subject to several publications and is out of scope for this paper (Adamala and Cidrin 2011; Fitzgerald 1998; Petter et al. 2012; Renkema and Berghout 1997; Walter and Spitta 2004). Mostly, ex ante evaluation and BI project justification in practice tend to rely on financial measurement approaches like cost/benefit analysis (Walter and Spitta 2004). Therefore, we use the same approach to measure the value of the regarded alternatives. We do not consider any options in the project, which excludes the real options approach as additional evaluation method. Efficiency measures like the ROI require reliable estimations of benefits, which is very hard to achieve in a BI context. A Method which is widely accepted for ex ante investment justification is the NPV. Considering the time value of money, the NPV method is always or almost always used by 74% of CFO’s in investment decisions (Graham and Harvey 2001). Therefore, easily applicable yet considering time effects, we prefer the NPV amongst the aforementioned financial evaluation methods. Although widely accepted, the NPV approach so far has not been applied to decisions on the PMM in BI projects.
Decision Model In the following we introduce a basic decision model to determine a project’s expected NPV depending on the applied PMM. Since the number of “brand name” PMMs has been estimated up to 1000 (Jayaratna 1994), the existing multitude of PMMs can hardly be evaluated one by one in this context. Therefore, introducing a classification is a first step to reduce complexity. All popular PMMs applied in IT projects can be assigned to the scale presented in Figure 1. Some PMMs – e. g. eXtreme Programming - may include characteristics of two adjoining types and hence be assigned very close to the boundary. This indicates that defining discrete categories within this continuous scale is challenging. We try to characterize and therefore distinguish four types of PMMS with common patterns: Agile development(A), Incremental development (I), Iterative development (S) and Plan-driven development (P). The currently hyped type referred to as “agile” provides a high degree of flexibility. However, the expression “agile” is used ambiguously in literature. To bring some clarity, we refer to the definition of Conboy (2009): ‘IS development method agility is the continual readiness to rapidly or inherently create change, proactively or reactively embrace change, and learn from change while contributing to perceived customer value (economy, quality, and simplicity), through its collective components and relationships
4
Thirty Fourth International Conference on Information Systems, Milan 2013
Otyepka et al. / Selecting Project Management Methodologies for Business Intelligence Projects
with its environment’. In contrast to A, P promises a high degree of documentation, comparability and planning reliability (Boehm and Turner 2003). Therefore, these two PMMs represent the extremes and illustrate the trade-off between flexibility and planning as indicated in the motivation section. Flexibility
Planning Agile (A)
Incremental (I)
Iterative (S)
Plan-driven (P)
e. g. eXtreme Programming; Crystal; Scrum
e. g. eXtreme Programming; Rational Unified Process (RUP)
e. g. Spiral Model
e. g. Waterfall Method; Cleanroom; SW-CMM; CMMI
Figure 1: Decision tree showing a project’s development structure for every period Iterative and incremental PMMs can be ranked in-between the named extremes, offering a certain degree of control and ex ante planning and the possibility to do rework when releases do not meet the customer’s needs. Although both have in common that they execute all project phases more than once (in contrast to the plan-driven approaches), they must not be used as synonyms. Incremental development I in contrast to S starts with a rough plan of the whole system in the first increment and seeks adding parts of the system in every following increment, comparable to a release-concept (Charvat 2003). This means that parts (releases) of the system are available for the customer early and frequent feedback after every increment is used to develop the system in a goal oriented way, until it fulfills the customer’s requirements. Should the project be abandoned for some reason, the customer can still use the so far developed parts of the system. Therefore the emphasis is on developing the important parts first and then adding to the project or expanding it (Charvat 2003). According to the common NPV approach, one compares the NPVs of two or more alternatives and chooses the alternative generating the highest one – given it is positive (Ross 1995). We assume the decision to run the BI project to be final, therefore the prerequisite of long-term project profitability is fulfilled. A redundant consideration of cash flows that are generated after the end of the project is not necessary since the PMM does not affect the long term profitability of the system in use. Thus, we avoid the risky estimation of mostly intangible or hardly measurable long term benefits and only consider cash flows during project runtime. Consequently, the resulting NPV is expected to be negative in our demonstration example. Validation is hard to achieve, since there is no empirical data including the same project run with different project management methodologies that could provide insights for future decision situations. At the same time it is not possible to provide identical pre-conditions to conduct an experiment for empirical purposes. Facing a lack of data history, we follow a design-oriented approach for developing the decision model. To validate the model we use a demonstration example with project cash flows approved by BI development experts in a first step. Hereby, we seek a better understanding of project management methodologies in general and their application in BI projects. In this research-in-progress paper, we take into account two risk parameters. The first refers to the abovementioned trade-off of planning and flexibility. Any project faces environmental changes or the need to refine customer requirements, which the applied PMM has to deal with. This is the major shortcoming associated with P, since it is widely criticized for being inappropriate to handle rapidly changing environments and customer requirements (Abrahamsson et al. 2009; Beck 1999; Charvat 2003; McCracken and Jackson 1982). I and S are more in the position to react on changes due to their release structure (Beck 1999) and A emphasizes close customer collaboration and is therefore even supposed to “embrace change” (Beck 1999; Boehm and Turner 2003; Conboy 2009; Fowler and Highsmith 2001). A, in contrast, is characterized as lightweight processes with close customer collaboration and short iterative cycles resulting in independently running modules (Boehm and Turner 2003; Fowler and
Thirty Fourth International Conference on Information Systems, Milan 2013
5
Project Management and IS Development
Highsmith 2001). The trend to value working software over documentation, individuals and interactions over processes and tools and responding to change over following a plan bears the risk of delivering perfectly working individual modules which are not thoroughly working as an integrated system (Black et al. 2009). If applying I or S, one can handle this requirement better since these PMMs include a higher level of planning activities, but following the prototyping approach, this risk is still considerable. Since P includes a high level of cross-organizational planning activities, this PMM bears the lowest risk of a lack of integration. Therefore, environmental dynamism δ and improper system integration εk are the two major risks to be taken into account in the proposed decision model. Environmental dynamism describes changes in a defined period of time within the project environment which require changes of the project’s management. Improper system integration results when parts of the system do not fit together at the end of the project and rework has to be done to fix the problem. Mostly, risk is perceived as negative or harmful deviation from the expected status. Since this is the most likely situation in BI projects, we only consider negative development in our decision model. Even if positive development is a possible scenario, this case is not necessary for the overall understanding of the model and might be subject to continuative research projects. The simplified decision tree presented in Figure 2 shows the possible development of a BI project. Like indicated before, we do not consider enhancing factors in a first step. In case a risk occurs, the decision of the company is to fix the failure (see A5).
t0
L= 1- δ
L= δ
t1
…
t1
…
Risk of environmental changes δ in every Period tn, n ϵ {1, …, T}
tn
L= 1- εk
L= εk
T
Risk-Neutral NPV See (Ia) below
T
Disturbing Factors See (Ib), (Ic) below
Risk of a lack of integration εk in the last Period T
Figure 2: Decision tree showing a project’s development structure for every period The underlying model needs more detailed explanation and is based on following formal assumptions: (A1) The decision to implement a BI system is final. The company only chooses between a preselected number of PMMs k K {A, I, S, P}. We assume that any chosen PMM is applied accurately. Due to its distinct strengths and weaknesses, each PMM has a different cost/benefit structure that is captured in aggregated cash flows per period. (A2) The model does not take into account the possibility of project abandonment in form of real options or the possibility of switching to another PMM during the project and can therefore be seen as a snap-shot of the currently expected situation. It is possible that repeated calculations at any point later during the project indicate that one of the (other) named actions might be beneficial. (A3) The project is characterized by two types of risks. These are the likelihood of environmental changes δ [0;1] within each project period and the likelihood of a lack of system integration εk [0;1] at the end of the project. We assume both risks to be independent. (A4) Since the periods are of equal length, δ can be assumed to be constant in every period and independent of the chosen PMM. εk is inherent to the chosen PMM and hence differs, denoted by index k. δ occurs in each period, εk only occurs in the last period T. (A5) If a risk occurs, the company will fix the failure. The costs associated with the correction are ascribed to the period the risk occurred. The costs to react on Dynamism, CDyn,k are involved
6
Thirty Fourth International Conference on Information Systems, Milan 2013
Otyepka et al. / Selecting Project Management Methodologies for Business Intelligence Projects
with δ and their amount depends on the regarded PMM. The cost for necessary rework due to a lack of system integration, CInt, are involved with εk. CInt can be assumed to be equal for any PMM.
Figure 3 can be transferred to the general model presented in Formula I:
NPVk ,risk adj NPVk ,exp ected *
C Int (1 i ) T 1 * C Dyn,k k * T (1 i ) * i (1 i ) T
(Ia)
(Ib)
(I)
(Ic)
Equation (Ia) denotes the common risk-free NPV calculated according to the project’s estimated cash flows. Equation (Ib) denotes the expected NPV of additional costs due to environmental changes which occur with the likelihood δ. Equation (Ic) denotes the expected NPV of additional costs coming along with the risk of improper system integration which we assume only to occur in the last period. (Ib) and (Ic) therefore represent disturbing factors which can downgrade the project’s NPV. To get the total expected NPVk,risk-adj, we calculate the sum of equations (Ia) to (Ic). Formula (I) allows us to evaluate a BI project depending on the applied PMM. In the next section we introduce a demonstration example to illustrate the model’s application and its benefits.
Demonstration Example For a better understanding of the proposed decision model, we apply it to a demonstration example according to Formula (I). The decision to run the BI-project is final, the estimation for project duration is 12 months with total costs of 2.000.000,00 USD. Seeking recommendation on the overall management approach, the preselection includes four types of development approaches namely A, I, S and P. Although practitioners have found P to be insufficient for BI-projects (Moss and Atre 2003; Moss 2009), we include it in our analysis for the sake of completeness. We do not narrow the selection down to particular methodologies like SCRUM, Capability Maturity Model (CMM) or others in this paper. This subsequent selection of a particular methodology might include qualitative parameters like company culture and project criticality (Boehm and Turner 2003), that are often included in non-value-based decision models. This is too far reaching to be addressed in this research-in-progress paper and might be matter of future research. Depending on the applied PMM, we split the estimated total costs of the project to the periods as shown in Figure 3.We assume total costs of 2,000,000.00 USD, split to the 12 periods and affecting net income in the period they occur. We assume the costs to be distributed according to the findings of Black et al. (2009). This means in case of applying A, we assume an iteration length of one month and equally high costs of $ 170.000,00 per period. In case of applying I, we assume a duration of each increment of 2 months and higher costs at the end of each increment. In case of applying S, we assume an iteration length of 3 months. Due to the fact that in iteration applies a short version of the plan driven development approach (Moss 2009), we assume slightly decreasing costs within every iteration (Black et al. 2009). Accordingly, we assume slightly decreasing costs across project duration in case of applying P. The interest rate of 1% per month, resulting in an annual interest rate of 12,7% is notably higher than the money market interest and makes allowance for risk averse stakeholders. The expected additional costs due to the risk structure approximate 20% of the estimated costs, a rate which is generally assumed by consultancies and varies slightly depending on the applied PMM.
Thirty Fourth International Conference on Information Systems, Milan 2013
7
Project Management and IS Development
t
Estimated Costs Agile
Increm ental
1
$1 7 0.000,00
2
$1 7 0.000,00
Spiral
Risk-Dependent Costs Agile
Plan-driv en
$1 40.000,00 $220.000,00 $230.000,00 Cdy n , per month $200.000,00 $1 7 0.000,00 $220.000,00 CInt , one-time $1 40.000,00 $1 1 0.000,00 $220.000,00 Likelihood for $200.000,00 $220.000,00 $21 0.000,00 $1 40.000,00 $1 7 0.000,00 $1 80.000,00 a lack of sy stem integration $200.000,00 $1 1 0.000,00 $1 7 0.000,00
$1 5.000,00
Increm ental
Spiral
Plan-driv en
$24.000,00 $30.000,00
$40.000,00
$400.000,00
3
$1 7 0.000,00
4
$1 7 0.000,00
5
$1 7 0.000,00
6
$1 7 0.000,00
7
$1 7 0.000,00
$1 40.000,00 $220.000,00 $1 7 0.000,00
T=
12
8
$1 7 0.000,00
$200.000,00 $1 7 0.000,00 $1 60.000,00
i=
0,01 per month
9
$1 7 0.000,00
$1 40.000,00 $1 1 0.000,00 $1 20.000,00
10
$1 7 0.000,00
$200.000,00 $220.000,00
$1 1 0.000,00
11
$1 7 0.000,00
$1 40.000,00 $1 7 0.000,00
$1 1 0.000,00
12
$1 30.000,00
$1 60.000,00 $1 1 0.000,00 $1 00.000,00
ε+0,1
ε+0,05
ε+0,025
ε
Figure 3: Not-discounted costs split to the project’s periods Figure 4 shows the calculation results for all parameter combinations of δ and εk in 0.1 steps and gives a recommendation for the most appropriate PMM type. δ: Likelihood of environmental changes 0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
0 0,1 0,2 0,3
ε: Likelihood of a lack of system integration
0,4 0,5 0,6 0,7
Incremen‐ Iterative tal Develop‐ Develop‐ ment (S) ment (I)
Agile Development (A)
0,8 0,9 1
(I)
Figure 4: Exemplary matrix showing calculation results The advantage of this approach is that risk parameters do not have to be determined ex ante. Using this matrix, a decision maker gets an idea of the overall situation and can decide according to the provided information. It shows that practitioners’ estimation on plan driven development in BI-projects is confirmed, since there is no parameter-combination that shows P to be most appropriate. A is preferred for very high likelihood of environmental changes. S and I is more appropriate in moderately changing environments. Although a purely agile approach has been claimed to be inappropriate for BI projects as well (Moss 2009), the calculation shows that an agile approach is most promising for the majority of parameter combinations. This might be an indicator that the ability to embrace change is of high importance for BI projects. The result also indicates that the risk of dynamism outweighs the risk of system integration. One reason for this observation might be the fact that a flexible and highly agile project environment enables project leaders to fix integration issues once they are identified.
Practical Implications, Limitations and Conclusion so far The presented model fulfils all six initially stated requirements mentioned in section 2. (1) It is applicable to a class of problems, namely to decisions on the PMM type in a BI project. (2) It contributes to the body of knowledge since no formal quantitative model on the decision between agile and plan-driven PMMs could be found in literature. (3) The underlying NPV approach is established and results are easily reproducible. (4) The model adds value to the decision process by raising the clarity in decision situations and therefore improving the decision itself. (5) Since estimating the project’s cash flows is necessary for project planning anyway, the model is easily applicable. Decision makers have to only estimate two additional risk parameters. (6) Relying on a NPV-based decision model, we propose to complement
8
Thirty Fourth International Conference on Information Systems, Milan 2013
Otyepka et al. / Selecting Project Management Methodologies for Business Intelligence Projects
decision strategies on a PMM in BI projects by a quantitative, value-based and future-oriented approach. Nevertheless, the proposed model is beset with limitations that need to be taken into account when applying it in project settings: The proposed model does not provide decision support on a specific PMM and is only able to distinguish types of PMMs. This classification however is a necessary first step which makes it possible to analyse the associated PMMs for the final decision. Furthermore, the proposed model does not take into account project risks which frequently threaten project success, such as unavailable or unwilling business representatives or unengaged business sponsors and a lack of understanding for the project situation. Also, deficient methodology application and usage is not considered in the presented model although this is an important reason for project failure. Implementation risks like determining a project’s goals and requirements come along with any PMM and project. Since this is not a factor for differentiating the PMMs in our decision model, we cannot consider them. These risks are out of scope of our work and have to be addressed separately. Moreover, the proposed model only takes into account two parameters which might neglect important intangible values and social effects. This limitation might be addressed in the future by integrating more parameters, for example the parameters proposed by Boehm and Turner (2003). Finally, the proposed model claims to offer a clear recommendation for a decision, yet is prone to errors of the estimated underlying cash flows. To strengthen the reliability of the decision model, we plan to address this shortcoming by conducting further sensitivity analyses to test the impact of estimation errors regarding both, cash flows and risk parameters. We further plan to validate the model in collaboration with BI consultants by applying the model in the beginning of BI projects parallel to the consultancy’s established approach. Comparing the results of the proposed decision model with the decisions made during the project might offer interesting results. The idea of using the presented model in order to uncover formal contradictions to bring more clarity and understanding to the decision problem on PMMs in BI projects might be helpful in other IT projects as well. Doing so, one still has to consider that the parameters used for BI projects might not be applicable to another kind of project (like SD projects). It is necessary to verify if the model parameters are applicable to other projects and to adjust them if needed. Despite these limitations, the proposed formal NPV model is one way to bring more transparency to the described decision situation. Although not all kind of project risks are addressed, first trying to understand the project environment and then applying a diligent selection of a PMM should help to avoid project failure that goes back to the application of an inappropriate PMM.
References Abrahamsson, P., Conboy, K., and Wang, X. 2009. "'Lots Done, More To Do': the Current State of Agile Systems Development Research," European Journal of Information Systems (18:4), pp. 281-284. Adamala, S., and Cidrin, L. 2011. "Key Success Factors in Business Intelligence," Journal of Intelligence Studies in Business (1:1), pp. 107-127. Atre, S. 2003. "The Top 10 Critical Challenges for Business Intelligence Success," http://www.computerworld.com/computerworld/records/images/pdf/BusIntellWPonline.pdf. Beck, K. 1999. "Embracing Change with Extreme Programming," IEEE Computer (10:32), pp. 70-77. Benaroch, M., and Kauffman, R. 1999. "A Case for Using Real Options Pricing Analysis to Evaluate Information Technology Project Investments," Information Systems Research (1:10), pp. 70-86. Black, S., Boca, P. P., Bowen, J. P., Gorman, J., and Hinchey, M. 2009. "Formal versus Agile: Survival of the Fittest," IEEE Computer (9:42), pp. 37-45. Boehm, B. W., and Turner, R. 2003. Balancing Agility and Discipline: A Guide for the Perplexed, Addison-Wesley Longman Publishing Co., Inc. Brealey, R. A., Myers, S. C., and Allen, F. 2011. Principles of Corporate Finance, McGraw-Hill/Irwin.
Thirty Fourth International Conference on Information Systems, Milan 2013
9
Project Management and IS Development
Carroll, J. 2003. "The Process of ISD Methodology Selection and Use: A Case Study." Proceedings of the 11th European Conference on Information Systems, ECIS , Naples, Italy, pp. 379-392. Charvat, J. 2003. Project Management Methodologies: Selecting, Implementing, and Supporting Methodologies and Processes for Projects, Hoboken, NJ: Wiley. Cockburn, A. 2000. "Selecting a Project's Methodology," IEEE Software (4:17), pp. 64-71. Coenenberg, A. G., Mattner, G. R., and Schultze, W. 2003. "Wertorientierte Steuerung: Anforderungen, Konzepte, Anwendungsprobleme," in Finanzwirtschaft, Kapitalmarkt und Banken, Andreas Rathgeber, H. Tebroke and M. Wallmeier (eds.) Stuttgart, Germany, Schäffer-Poeschel, pp. 1-24. Conboy, K. 2009. "Agility from First Principles: Reconstructing the Concept of Agility in Information Systems Development," Information Systems Research (3:20), pp. 329-354. Copeland, T. E., Weston, J. F., and Shastri, K. 2005. Financial Theory And Corporate Policy, London, UK, Pearson. Fitzgerald, G. 1998. "Evaluating Information Systems Projects: A Multidimensional Approach," Journal of Information Technology (1:13), pp. 15-27. Fowler, M., and Highsmith, J. 2001. "The Agile Manifesto," Software Development (8:9), pp. 28-35. Fowler, M. 2005. "The New Methodology," http://www.martinfowler.com/articles/newMethodology.html. Gartner, Inc., Stamford, Connecticut 2012a. "Fewer than 30 Percent of Business Intelligence Initiatives will Align Analytic Metrics Completely with Enterprise Business Drivers by 2014," http://www.gartner.com/it/page.jsp?id=1891515.Gartner Research Gartner, Inc., Stamford, Connecticut 2012b. "Worldwide Spending on Enterprise Application Software to Increase 4.5 Percent in 2012 ," http://www.gartner.com/it/page.jsp?id=2055215. Gartner Research Graham, J. R., and Harvey, C. R. 2001. "The Theory and Practice of Corporate Finance: Evidence from the Field," Journal of Financial Economics (60), pp. 187-243. Hardy-Vallee, B. 2012. "The Cost of Bad Project Management," http://businessjournal.gallup.com/content/152429/cost-bad-project-management.aspx#2. Höhn, R., and Höppner, S. 2008. Das V-Modell XT: Grundlagen, Methodik und Anwendungen, Heidelberg,Germany, Springer. Jayaratna, N. 1994. Understanding and Evaluating Methodologies: NIMSAD, a Systematic Framework, New York City, USA, McGraw-Hill, Inc. Kimball, R., and Ross, M. 2002. The Data Warehouse Toolkit: The Complete Guide to Dimensional Modeling 2002, Hoboken, NJ: Wiley. Mahmood, M. A. 1987. "System Development Methods—a Comparative Investigation," MIS Quarterly (3:11), pp. 293-311. McAfee, A. 2006. "Mastering the Three Worlds of Information Technology," Harvard Business Review. (11:84), pp. 141. McCracken, D. D., and Jackson, M. A. 1982. "Life Cycle Concept Considered Harmful," ACM SIGSOFT Software Engineering Notes (2:7), pp. 29-32.
10 Thirty Fourth International Conference on Information Systems, Milan 2013
Otyepka et al. / Selecting Project Management Methodologies for Business Intelligence Projects
McHugh, O., and Hogan, M. 2011. "Investigating the Rationale for Adopting an InternationallyRecognised Project Management Methodology in Ireland: The view of the Project Manager," International Journal of Project Management. (5:29), pp. 637-646. Mertens, P. 1999. "Operiert die Wirtschaftsinformatik mit den falschen Unternehmenszielen? - 15 Thesen," in Wirtschaftsinformatik und Wissenschaftstheorie, Jörg Becker, W. König, R. Schütte, O. Wendt and S. Zelewski (eds.) Wiesbaden, Germany, Gabler, pp. 379-392. Mikulenas, G., and Kapocius, K. 2011. "An Approach for Prioritizing Agile Practices for Adaptation," Information Systems Development, pp. 485-498. Moss, L. T., and Atre, S. 2003. Business Intelligence Roadmap: The Complete Project Lifecycle for Decision-Support Applications, Boston, USA, Addison-Wesley Professional. Moss, L. T. 2009. "Beware of Scrum Fanatics On DW/BI Projects,"Enterprise Information Management Insight MAGAZINE, (3:3). Naumann, J. D., and Palvia, S. 1982. "A selection Model for Systems Development Tools," MIS Quarterly pp. 39-48. Österle, H., Becker, J., Frank, U., Hess, T., Karagiannis, D., Krcmar, H., Loos, P., Mertens, P., Oberweis, A., and Sinz, E. 2011. "Memorandum on Design-Oriented Information Systems Research," European Journal of Information Systems (1:20), pp. 7-10. Petter, S., DeLone, W., and McLean, E. R. 2012. "The Past, Present, and Future of “IS Success”," Journal of the Association for Information Systems (5:13), pp. 341-362. Renkema, T. J. W., and Berghout, E. W. 1997. "Methodologies for Information Systems Investment Evaluation at the Proposal Stage: A Comparative Review," Information and Software Technology (1:39), pp. 1-13. Ross, S. A. 1995. "Uses, Abuses, and Alternatives to the Net-Present-Value Rule," Financial Management (24:3), pp. 96-102. Saarinen, T. 1990. "System Development Methodology and Project Success: An Assessment of Situational Approaches," Information & Management (3:19), pp. 183-193. Verhoef, C. 2005. "Quantifying the Value of IT-Investments," Science of Computer Programming (3:56), pp. 315-342. Walter, S. G., and Spitta, T. 2004. "Approaches to the Ex-Ante Evaluation of Investments into Information Systems," WIRTSCHAFTSINFORMATIK (3:46), pp. 171-180. Watson, H. J. 2009. "Tutorial Business Intelligence - Past, Presence, and Future," Communications of the AIS (1:25), pp. 487-510. Wells, H. 2012. "How Effective Are Project Management Methodologies? An Explorative Evaluation of Their Benefits in Practice," Project Management Jounal (6:43), pp. 43-58. Williams, S., and Williams, N. 2003. "The Business Value of Business Intelligence," Journal of Data Warehousing (4:8), pp. 30-39.
Thirty Fourth International Conference on Information Systems, Milan 2013
11
