Estimating Effort and Cost in Software Projects − ISBSG A Multi-Organizational Project Data Repository for Project Estimation And Benchmarking Briefing Centre for Software Process Technologies University of Ulster Northern Ireland 18 September 2003 Pierre Bourque École de technologie supérieure Montreal, Canada
Cumulative Concern Value
109
100 93
S/ w Pr oc re wo es sA rk dh er en ce Pr oj ec tP lan ni De En ng ve su lop rin g ing qu Re ali ty qu M ire an m ag en ing ts Re qu ire m en ts M ain Te ta in am in sc gs om /w m u Te nic ch ati no on log yc ha ng es M an ag ing fau lts
Es tim ati ng Ta Ri sk sk s m an ag em en t Pr od uc tiv W Pr ity or o j ec kp tT ra rac cti se kin co g ns ist en cy
Benchmarking the NI software industry Concern measures for Engineering Aspects
120 106
99 91 88 85 84 83
80
Centre for Software Process Technologies
81 79 74 69 65
60 49
40
20
0
Engineering Aspect
2
Ice Bag
ISBSG Centre for Software Process Technologies
3
École de technologie supérieure ETS • An Engineering University – Within the network of the Université du Québec • In + 10 cities across Québec
• Engineering Specialties: – – – – – – –
Aeronautical Engineering Automated Manufacturing Construction Engineering Electrical Engineering Mechanical Engineering Software Engineering Telecommunications + Information Technologies Centre for Software Process Technologies
4
ETS Software Engineering Lab. Strengths • Recognized world-wide in software engineering for: – Building consensus in software engineering – Leadership of world-wide initiatives – Strong applied research focus
• www.lrgl.uqam.ca
Centre for Software Process Technologies
5
Software Engineering Management • Sample of R & D topics: – Guide to the Software Engineering Body of Knowledge (SWEBOK) – Second generation of functional size method: COSMIC – Estimation models – Quality Engineering – Risk of measurement programs – Metrology
Centre for Software Process Technologies
6
Agenda • • • • •
Principles of Credible Estimation Overview of Software Functional Sizing Overview of ISBSG Overview of the Repository An example of using ISBSG for Duration Estimation • Conclusion Centre for Software Process Technologies
7
How do you build your estimates? • • • • • • • • •
How do you build estimates in your organization? What are the inputs to your estimates? Is your estimation process documented? Do you collect and use historical data? Do you collect data that is never used? What is the reliability of this data? Do you track your estimates? How do you size the amount of work to be done? Are the quality of your estimates very much based on the competency of a few key people in your organization? Centre for Software Process Technologies
8
What is software engineering ? • IEEE 610.12: (1) The application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software; that is, the application of engineering to software. (2) The study of approaches as in (1).
Centre for Software Process Technologies
9
Many Commercial Estimation Tools on the Market • Many well-known commercial tools: – – – – – •
Checkpoint Project Workbench PQM Plus SLIM …
See O. Mendes, A. Abran and P. Bourque. Function Point Tool Market Survey. Université du Québec à Montréal,1996 Available at www.lrgl.uqam.ca/publications/pdf/204.pdf Centre for Software Process Technologies
10
What is the result? ¤ Lots of numbers, graphs, nice slide presentation, ¤ Industry data often not verifiable, ¤ “Black box” approach, ¤ Bottom line: everybody sees what they want to see...
Centre for Software Process Technologies
11
Mixed PUBLISHED Results... • Panoply of software cost models • Several studies have been conducted to assess the accuracy of these models on various databases • However, no study has proven the superiority of any models excepted for limited applications • Often small data samples Centre for Software Process Technologies
12
Underlying Principles of CREDIBLE Estimation - 1 As Defined in Park et al. (94): « Estimates are made by people, not by models. They require reasoned judgments and commitments to organizational goals that cannot be delegated to any automated process ». Centre for Software Process Technologies
13
Underlying Principles of CREDIBLE Estimation-2 « All estimates are based on comparisons. When people estimate, they evaluate how something is like, and how something is unlike, things that they or others have seen before ». Centre for Software Process Technologies
14
Underlying Principles of CREDIBLE Estimation-3 « Before people can estimate, they must acquire knowledge. They must collect and quantify information from other projects, so that they can place their comparative evaluations on demonstrably sound footings ». Centre for Software Process Technologies
15
These Principles Imply: • To be credible, an estimation process must inherently be white-box • Software project estimation which has been plaguing the industry for years can only be solved through a cooperative data collection effort • Much research still has to be done Centre for Software Process Technologies
16
Agenda • • • • •
Principles of Credible Estimation Overview of Software Functional Sizing Overview of ISBSG Overview of the Repository An example of using ISBSG for Duration Estimation • Conclusion Centre for Software Process Technologies
17
Size of what …
Context...
HOW BIG IS IT ?
Project Size The total effort, estimated or actual in work-hours or staffmonths Software size the size of the requirements (functions) or of the deliverables (modules, lines of code) Centre for Software Process Technologies
18
FUNCTIONAL
HOW BIG IS IT ?
TECHNICAL
Software size measurement Mmm… so many programs, so many lines of code... • Meaningful to the technical staff,
• Meaningless to management, • Poor portability, • Only known precisely when too late to use Mmm… so much functionality delivered to the users... • Meaningful to management,
• Meaningful to technical staff, • Portable, • Can be measured early on, • Must be independent from effort, method or technology
Centre for Software Process Technologies
19
The ‘Functional Size’ of software
Ø ISO/IEC/JTC1/SC7
Standard #14143
definition: “ Functional Size : A size of software derived by quantifying the functional user requirements”
An analogy...
2000 sq. ft. 4000 sq. ft.
Software Functionality
500 cfsu
Software Functionality
Centre for Software Process Technologies
1000 cfsu
21
Very Significant Amount of Work on Software Functional Size?” ISO ‘FSM’ Standard MkII FPA 1.3
3-D FP’s MkII FPA Feature Points Allan Albrecht FPA
1980
Full FP’s V.1
IFPUG 4.1
IFPUG 4.0
1985for Software Process 1990 Centre Technologies
COSMIC FFP V. 2
1995
2000 22
Usages of Software Sizing • • • •
Estimation Benchmarking Productivity Trend Analysis Contract Payment Mechanisms – Development – Corrective Maintenance and Support
• Quality Tracking Centre for Software Process Technologies
23
Productivity Trend Analysis FP Unit Costs
Hours/Extended FP
12 10 8 6 4 2 0 1990
40
1991
1992
1993
1994
1993
1994
Year 12
30
Extended FP
Effort
10 20
10
0 1990
1991
1992
1993
8 6 4 2
1994
Year
0
Centre for Software Process Technologies
1990
1991
1992
Year
24
Agenda • • • • •
Principles of Credible Estimation Overview of Software Functional Sizing Overview of ISBSG Overview of the Repository An example of using ISBSG for Duration Estimation • Conclusion Centre for Software Process Technologies
25
ISBSG Mission • “To help improve the management of IT resources by both business and government through the provision and exploitation of a public repository of software engineering knowledge which is standardized, verified, recent and representative of current technologies.” Centre for Software Process Technologies
26
ISBSG The 1st. Phase... Australian Software Metrics Association, (ASMA) • Motivation – practitioners • wanting control • seeking best practices
• Definition of Productivity – Project Delivery Rate = “Hours per Function Point”
Centre for Software Process Technologies
27
ISBSG The 2nd. Phase ...the ASMA Work • First Release of data (24 projects) in December 1992 • Revised standard & collection package several times • 6 Releases of Data up to June 1994
Centre for Software Process Technologies
28
ISBSG The 3rd. Phase Became the… International Software Benchmarking Standards Group ISBSG www.isbsg.org Centre for Software Process Technologies
29
International Membership Current membership: • Australia, Finland, Germany • India, Italy, Japan, Netherlands, • Spain, Switzerland, • United Kingdom, USA
Centre for Software Process Technologies
30
Principles of ISBSG • Principles of Evolution – – – – –
independence data integrity anonymity practitioner driven practitioner accessible
Centre for Software Process Technologies
31
Initial Principles of Evolution • “ Any project team anywhere in the world may, for insignificant cost, benchmark themselves against world’s best”
Centre for Software Process Technologies
32
ISBSG Existing Standard • Collection Philosophy • complete in 30 mins • meaningful, available, objective
• Project Attributes – – – –
Functional Size Work Effort, Team Size, language Cost Quality......Etc........... Centre for Software Process Technologies
33
Collection Process • • • • • • •
Security Procedures Validation Procedures Repository entry Project Benchmark Report Rating comments Re-submission Re-rating and project update Centre for Software Process Technologies
34
Summary - ISBSG Strengths • • • • •
Not profit motivated Provides Project Benchmarking Allows direct access to the data Allows networking Broad representation of IT – technologies, organisation types, geography
• International based standard Centre for Software Process Technologies
35
Agenda • • • • •
Principles of Credible Estimation Overview of Software Functional Sizing Overview of ISBSG Overview of the Repository An example of using ISBSG for Duration Estimation • Conclusion Centre for Software Process Technologies
36
ISBSG Release 8 • Demonstration of ISBSG Release 8 Data Set
Centre for Software Process Technologies
37
Age of Projects (Data from Release 7) REPOSITORY PROJECTS
year not known 1989
PROJECTS
1990
250
1991
200
1993
1992
1994
150
1995
100
1996
50
1998
1997
1999
0
2000
YEAR Centre for Software Process Technologies
2001
38
Projects by Country Australia Japan United States Netherlands Canada United Kingdom India France Brazil Other 0
50
100
150
200
250
300
Number of projects
Centre for Software Process Technologies
39
Projects by Organization Type Communication Government, Public Administration Financial, Property & Business Services Insurance Manufacturing Banking Electricity, Gas, Water Wholesale & Retail Trade Computers, ISP Ordering Billing Energy, oil, mining Voice provisioning Engineering Other 0 50 100 150 200 250 300 350 400 450 500 Centre for Software Process Number of projects Technologies
40
Projects by Business Area Telecommunications Insurance, actuarial Banking Financial (excluding Banking) Manufacturing Accounting Engineering Sales & Marketing Inventory Legal Personnel Research & Development Logistics Other 0
50
100
150
200
Number of projects Centre for Software Process Technologies
250
300
350 41
Types of Projects Enhancement
New development
Re-development
0
200
400
600
800
1000
1200
Number of projects
Centre for Software Process Technologies
42
Application Type Transaction/Production System Management Information System Office Information System Network Management Billing Ordering Electronic Data Interchange Process control, sensor control, real time Decision Support System Executive Information System Voice provisioning Knowledge Based System Other
0
100
200
300
400
Number of projects Centre for Software Process Technologies
500
600
43
Use of Development Techniques Data modelling Process modelling Business area modelling Event modelling Joint App'n Development Rapid App'n Development Object Oriented Analysis Object Oriented Design Prototyping Multifunction teams Regression testing Timeboxing Standards (ISO 9000; CMM) 0
50 100 150 200 250 300 350 400 450 Centre for Software Process Number of projects Technologies
44
The ISBSG Repository Positioning • Probably represents top 20% of industry • Primarily MIS Applications
Centre for Software Process Technologies
45
Agenda • • • • •
Principles of Credible Estimation Overview of Software Functional Sizing Overview of ISBSG Overview of the Repository An example of using ISBSG for Duration Estimation • Conclusion Centre for Software Process Technologies
46
Strategic Importance of Time-to-Market • Project manager’s dream: – – – –
Complete and stable product requirements High quality Low costs Short time-to-market
• Time to market or project duration is often the hardest one to pin down • A variation of the other three have a determining effect on it Centre for Software Process Technologies
47
Adopted Viewpoint in this Research Product Requirements
Product Size
Project Duration
Project Effort Adopted Viewpoint
Centre for Software Process Technologies
48
2. Selecting a Data Sample 2.1
ISBSG release 4 (1997)
2.2
Basic selection criteria
2.3
Distribution analysis
•
Effort
•
Duration
•
Summary Centre for Software Process Technologies
49
2.1 ISBSG Release 4 • Release 4 (1997) contains 396 completed projects – Contribution from 13 countries – 2/3 new development, 1/3 enhancements & redevelopment – 34% Txn proc., 38% MIS, 14% office information – 3/4 developped in-house for internal use – 67% Mainframe – 46% 3GL, 38% 4GL
Centre for Software Process Technologies
50
2.2 Basic Selection Criteria • No reasonable doubts on data validity according to ISBSG screening • Known effort, known duration and known platform
ä
Duration (D) in calendar months
Effort (E) in personhours
Number of observations (n)
312
312
Minimum value
1,0
10
Maximum value
78,0
106 480
Mean value
10,5
5 933
Standard deviation
9,0
12 169
Median
8,0
2 228
D range: 1 to 78 months E range: 0,1 to 761 person-months
312 projects satisfied all criteria Centre for Software Process Technologies
51
2.3 Basic Criteria Scatter plot of effort vs. duration (n=312) months 80 70
DURATION
60 50 40 30 20 10 0 0
20000
40000
60000
80000
100000
person-hours
EFFORT
Centre for Software Process Technologies
52
2.3 Distribution Analysis - Effort No transform
Value Significance (α