Initial Effects of Software Process Improvement on ... - Semantic Scholar
Proceedings of the 29th Annual Hawaii International Conferenceon SystemSciences- 1996
Initial Effects of Software Process Improvement on an Experienced Software Development Team Eugene G. McGuire Computer Science and Information Systems The American University Washington, D.C. 200164116
Abstract
(as judged by customer requirements) software products. This team is now being required to implement new processes and procedures with the goal, as specified by organizational management in adopting the CMM as their model, of institutionalizing quality and reliability in their software systems. The issue being examined is the effect of the organizationally mandated software process improvement initiative on the group dynamics and team effectiveness of the existing software development team. It was anticipated by organizational management that the past success of this team would contribute to their perceptions of and reactions to the software process improvement effort. Specifically, it was expected that the software development team would prefer the previous (and proven -- in their eyes) development environment over the new (and unproven) development environment which would result from undertaking the software process improvement effort using the CMM. Organizational management hoped to facilitate this potentially difficult transition period by monitoring the perceptions and reactions of the software development team members during the initial stages of the process improvement effort and then providing training where needed. By capturing the team’s perceptions of their current strengths and their observations of change-related issues and current change management strategies employed by the organization, organizational management hoped to gain insight into the team’s attitudes (positive and negative) towards the software process improvement effort. Organizational management would then provide training in the form of workshops or seminars on issues identified as potentially troublesome or simply provide information sessions on issues that appeared to be not clearly understood by the team. For example, as expected the initial survey taken by the team showed few strengths in process focus.
Zhis paper discusses the initial stages of a long-term case study designed to examine the eflorts of an experienced software development team in moving to a more process-driven software development environment. This team has previously produced sojiware that has met functional, schedule, and cost criteria as specified by their customers. The sofiare development manager, as directed by an organizational mandate, has initiated efforts to move this team into a more structured, formalism-based development and testing environment. The Capability Maturity Model (CMM) developed by the Software Engineering Institute (SEI) is being used as the model for this effort. The group dynamic and team development aspects of this e#ort are being carefilly monitored to determine possible sources of resistance to change and to develop intervention “just-in-time” training sessions that can address identif?ed problem areas, particularly those that may directly affect productivity, quality, andschedule. This paper discussesinitialfindings in this area and addresses them within the CMM framework.
1.
Introduction
Software development teams, because of their often mission-critical status, are increasingly in the forefront of change processes in an organization. Accompanying the increased reliance on effective software teams in general is a growing emphasis on quality products and services that are often produced through rigorous accountability procedures designed to standardize process and quantify data measurement [13,18,21]. This research study examines the effects of a software process improvement effort on an experienced software development team that has previously produced successful
1060-3425/96 $5.0001996IEEE
713
Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE
Proceedings of the 29th Annual Hawaii International
Literature Review
2.1
Software Process Improvement
1996
under the CMM framework), managers use metrics to quantitatively measure the quality of a software product. Schedules and budgets are based on historical performance and are realistic; the predicted results for product cost, schedule, functionality, and quality are routinely achieved. Based on the predictability of the project results, reliable management decisions can be made about tradeoffs among these outcomes. Models such as the CMM are designed to contribute toward lower costs and reduced intervals because they specify high quality processes. These processes are capable of predictable results (producing high quality software products) which in turn help with the planning and management of projects.
Organizational management was then able to justify and able to institute a series of workshops designed to explain in detail how specific software processes could be improved in that environment. This went far beyond the initial introduction to CMM training that had been originally planned for the team. One example of improving information flow resulted from the team’s perception that employee efforts to change were not well recognized by the organization (see Table 2). As a result of this feedback, organizational management held an information session with the team that focused on rewarding positive efforts and recognizing role models. Some of the questions this research attempts to address include examining whether the volatility of internal software engineering efforts can be affected by: team structure and norms of operation, integration of process into team responsibilities, and control activities coordination with management in team development and responsibility. This research can potentially benefit organizational managers and software development project managers as they attempt to facilitate a smooth transition from a nonprocess driven software development environment to a process-focused environment under the CMM.
2.
Conference on System Sciences -
2.2
Managing People and Teams
Management can also more clearly define relationships concerning roles and responsibilities between software developers and users. This approach recognizes the importance of communication between customer and software developer as well as between professionals with different skills. For example, it is inevitable that teams of IS and functional area professionals formed to create complex information systems bring distinctive varieties of human behavior and learning which may deeply affect the resulting system to any complex development effort. This diversity in style occurs despite structure imposed by task requirements, group behavior norms, and extemallydeveloped guidelines such as the CMM. Nevertheless, it is expected that by establishing and enforcing formal procedures for communication and collaboration as indicated under the CMM, team performance can be enhanced and problems diminished. Another aim of this approach is to move the emphasis of the development from being an artistic expression of individuals working mostly alone to one of team-based engineering. This is not an easy task as some research [7,8,9] shows that the single most important motivator for data processing professionals is the creativity of the job itself. In addition, self-managed teams appear to be evolving at a rapid pace as a result of TQM and reengineering initiatives in many organizations. The literature shows many studies which demonstrate that extraordinary productivity results when people are placed in selfmanaged teams in highly constrained environments. There is a substantial literature relating tales of “heroic” operations working against impossible deadlines with minimal resources, thereby becoming extraordinarily motivated and sometimes flouting the expectations of the mature parent organization. Some of these projects were poorly structured but, nevertheless, succeeded as a result of the personalities of the leaders [19,33].
Software process improvement arose out of the quality principles of Deming, Juran, and Crosby in the mid-1980s in response to the belief that better management of software development processes would lead to improved software reliability and quality. One of the better recognized models of software process maturity is the Capability Maturity Model (CMM) developed by the Software Engineering Institute (SEI) at Carnegie Mellon University in Pittsburgh [ 11,411. Software process improvement as typified by the CMM provides guidelines for an organization to employ a structured approach to strategically improve all aspects of a software development operation and dramatically changes or extends concepts previously employed under recognized systems development models such as the waterfall and spiral approach [3,46]. Organizations previously would often attempt to improve performance in an ad-hoc manner by using technology driven approaches instead of addressing root problems. For example, organizations would often assign more staff or more tools to an already late and over-budget development project rather than addressing the reasons why the project had problems. In contrast, in a mature organization (as measured
714
Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE
Proceedings of the 29th Annual Hawaii International Conferenceon SystemSciences- 1996 members are not well defined. Project and design information is not stored or communicated effectively. The organization does not consistently apply good engineering management to the software development process. Descriptors such as “ad-hoc” and “chaotic” have been frequently applied to this level. Early assessments done by the SE1 found that more than 70% of assessed organizations were at Level 1. Level 2 -- Repeatable: The organization has achieved a stable process with a repeatable management control level by initiating rigorous project management of commitments, costs, schedules, and changes. The organization uses standard practices for managing development activities including: staff estimating; cost estimating; scheduling; change and version control on and design data; and requirements, specifications, managerial reviews. The managers are trained in these management practices and software engineers are trained on the use of tools and procedures. The organization is good at doing similar types of work but is challenged by new types of work because it lacks an orderly framework for improvement. Level 3 -- Defined: The organization has defined the process as a basis for consistent implementation and better understanding. Goals, objectives and strategies are developed and promulgated. The software development organization is multi-disciplinary and a Software Engineering Process Group (SEPG) is established to lead process improvement. At this point, the risk of introducing advanced technology is greatly reduced. Level 4 -- Managed: The organization has initiated comprehensive process measurements and analysis. This is when the most significant quality improvements begin. The senior managers are involved in the product and process management. The performance of the design process is monitored and measured; data gathering is repeatable and auditable. The organization typically bases its operating decisions on quantitative process data, and conducts extensive analysis of the data gathered during reviews and tests. Early assessments conducted by SE1 found no organizations operating at this level and recent assessments show only a few at this level. Level 5 -- Optimized: The organization now has a foundation for continuously improving and optimizing the process. The organization conducts analyses of the metrics data gathered during the design process and use these for planning improvements. They focus on improving and optimizing the process and products by establishing improvement goals for quality, productivity, and cost. Early assessments conducted by SE1 found no organizations operating at this level and recent assessments show only a very few at this level. Two variations on the CMM are the Systems
Much professional literature exists that attempts to predict the future of the information systems discipline, especially in relationship to managing people and teams involved in complex projects [5,14,17,23]. A closely related stream of research literature focuses on the dominant information technology issues of the 1990s [40] and work performance issues of software developers as they attempt to cope with complex systems development projects [28,44]. Common themes include: systems developers rapidly approaching their limits in terms of handling complexity, organizational managers increasingly disappointed by the lack of contribution of information systems to profits, the chaos of personal computing, and the overall failure of existing approaches for developing successful systems. Indeed, as complexity grows, practitioners and academics alike are examining and revising educational requirements for software engineers [l&32,42].
2.3
Characteristics of Team-Based Process Maturity
Part of the reason for the recent attention to quality in software development is that empirical research consistently shows that many software development projects suffer from a lack of proven and well established methods [16]. Models such as the CMM are only now advancing beyond the embryonic stage and being accepted by a wide variety of organizations [25]. They still have not yet undergone the rigorous test of years of industrial application. In fact, many organizations that begin assessments, do not follow through in improving their development processes [39]. Many organizations, nevertheless, employ talented software engineers and managers and produce successful software products even though they are not operating with a strong process focus [12,25]. In such organizations, it is appropriate to examine if team and management issues arise as those software development groups move towards a more structured, process-oriented environment. The CMM requires that organizational management, users, customers, and software development professionals communicate and collaborate frequently and effectively to achieve quality and repeatable processes. In fact, team training is specifically required to fully satisfy the requirements of some key process areas in the CMM. The CMM breaks down the software engineering capabilities of organizations into 5 maturity levels from Level 1 (Initial) to Level 5 (Optimizing). Generally, the levels can be characterized and distinguished in terms of team and management processes as: Level 1 -- Initial: There are no well-defined procedures and management. Roles of the design team
715
Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE
Proceedings of the 29th Annual Hawaii International
related to the existing software development environment. This survey was conducted to determine the perceived current (or pre-process improvement) status of these factors. Seventy-nine factors were identified from a literature review and categorized under the categories of: Teamwork Behaviors, Team Team Background, Leadership, Change Management, Quality Focus, Process Project Management, Focus, Customer Focus, Methodologies, Metrics, and Tools. These factors were defined and discussed with team members prior to the survey. Survey participants were asked to rate the current status of the 79 factors on a Likert scale. Second, team members were surveyed two months later on their perceptions concerning the extent to which various change-related issues (again identified through a literature review) were encountered during the initial stages of the process improvement effort as the team was making the transition from an old to a new way of operating. The Likert scale survey measured the team members’ perceptions of the frequency with which 20 typical organizational change issues had arisen. Third, at the same time, they were asked to evaluate the extent to which they observed various change management strategies being employed by the organization and their managers during the transition period from an “ad-hoc” development environment to a more structured development environment. This survey of 20 typical change management strategies was also done with a Likert scale. Results from these three initial surveys were used by organizational management and software development management to develop and provide training and information sessions for team members and management in the software development environment. Fourth, team members were surveyed six months after the software process improvement effort was initiated (four months after they identified change-related issues and received training related to these issues) on the same set of 79 factors related to the software development environment that they addressed in the first survey. This survey was conducted to determine the perceived status of these factors after the initial stage of the process improvement effort was underway. A comparison between the responses of the team’from the first to the last survey in this study reveals their perceptions of the differences in the status or relative maturity of the factors in the survey after the software process was underway. This research employs Likert-scale surveys and is intended to capture the perceptions of the respondents. It is recognized that perceptions can be formed by many factors and that each individual respondent brings their own experience and interpretation to each item in the survey. These are well-recognized limitations of Likert-
Engineering Capability Maturity Model (SE-CMM), Version 1.0 (December 1994) [l] and the People Management Capability Maturity Model (P-CMM), Version 0.3 (April, 1995) [lo]. These variations of the original CMM seek to focus the organization’s resources on the whole systems engineering process and the human resources aspects of the processes respectively. By extending the initial CMM framework to cover systems and engineering and human resource factors the SE1 is attempting to address problems that fall outside the software domain. Both of these new models, particularly the I’-CMM include substantive discussion of how people, technical and end-user, are instrumental in the production of quality software/systems. Most improvement programs to date have emphasized process or technology and not people. People management practices in many organizations, despite a significant amount of literature addressing people-related issues [34,36,43,44,48,50], do not address people issues in a systematic and structured manner. In addition, most managers are untrained or inadequately trained in implementing corrective solutions once people problems are identified [45,49,53]. Also, organizational factors, while widely cited [6,37,38,47] are often not systematically analyzed for their effect on process improvement efforts. The characteristics of the SE1 levels show that information systems professionals who work on software engineering projects must be capable of being highly productive in complex, team-oriented environments with strong emphases on process control and overall quality [51,54]. These requirements are critical but may not always be present in current information systems professionals. The lack of appropriate team and process control skills can greatly contribute to internal organizational volatility. This research study attempts to address these issues.
3.
Conference on System Sciences - 1996
Research Methodology
Sixty four software development professional comprised the team that is the subject of this study. The team included project managers, analysts, programmers, testers, configuration management specialists, and documentation specialists. They worked on separate, but related, modules of the same project. Some of their interactions were within the module groups and some were within the larger team. Team members had on average approximately two years of experience woking in this team. This research study was conducted by first surveying the members of the software development team on their perceptions of the strength of a wide variety of factors
716
Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE
Proceedings of the 29th Annual Hawaii International Conference on System Sciences - 1996 scale surveys and therefore care should be taken in generalizing these results to other teams and/or organizations. The advantages of Liiert-scale surveys, however, are that they are a means for quick and efficient data collection and can indicate trends in the thinking of the respondents as a whole. Additional insights and richer detail can be obtained through other research methods such as interviews and will be part of future stages of this long-term study. The value of this particular research study and the methodology is in the process employed as opposed to the specific results of the surveys. The results obtained from surveys such as these will obviously vary depending on the organizational and software development environment. The specific results from the surveys in this study, therefore, may not be generalizable to other organizations. However, the process of utilizing Likert-type surveys to take the temperature of the team regarding project management and process management issues is valuable if the organization is committed to responding to this feedback to facilitate a smooth transition to a processdriven software development environment.
4.
Data Results
4.1
Survey number 1 & 4 -- Pre-process improvement status and in-process improvement status
Change Management: organization communicates goals; organization communicates values; negotiation used to obtain agreement and cooperation; behavior supporting change rewarded; resources provided/redirected to support change; and increased participation and decision making encouraged. Quality Focus: pre-delivery defect reports; test status reports; project post-mortems; quality plan; defect distribution through code; quality assurance function; root analysis; cause quality assurance reports; review/inspection defect removal efficiency; and errorprone module analysis. ProcessFocus: software process definitions; software process monitoring; software process assessment; software process integrated with organizational objectives; and software process practices reflect state-of-the-art. Customer Focus: customer support; customer defect reports; customer requests for enhancement; customer satisfaction surveys; customer involvement during acceptance testing; customer acceptance testing; customer satisfaction with documentation; and customer involvement during customer documentation. Project Management:project/development plan; actual vs. estimated schedule; milestone and progress reviews; software estimation methods and tools; project management audit; schedule estimation process effectiveness; review/inspection scheduling; effort estimation process effectiveness; and product plan. Methodologies: requirements review; functional (high level) design; system test plan; testing function; system testing; software development processes; system test plan review; data dictionary; software architecture; reuse; reengineering; and prototyping. Metrics: defect quantities; defect severities; response time; throughput; life cycle productivity measurements; and reused code size. Tools: program debugging tools; support software effectiveness; internal documentation tool effectiveness; and source code project library. A five-point Likert scale was used with the following values: 1 = None; 2 = Weak; 3 = Average; 4 = Strong; 5 = Excellent. Results from the first and last surveys are summarized in the following table. The mean score for all the factors in each category has been averaged for pre-process improvement (before --beginning of research study) and in-process improvement (during -after 6 months). The mean scores for each of the items in the 11 categories in Table 1 were captured in the data collection phase of this study and can be used by organizational management and software development project management to identify specific weaknesses in each of these categories. Those details are not presented here.
At the beginning of this research study (pre-process improvement) and after six months of process improvement efforts survey participants were asked the following question: “In your opinion and based on your experience in your current software development team, how ‘would you rate the following in your team:” Respondents were presented with 11 categories of project and process related factors. There were 79 total factors. The categories and factors in those categories are: Team Background: project team experience; programming language experience; application experience; project management experience; software engineering training for staff; review and inspection experience; and training plan. Teamwork Behaviors: clearly defined roles and responsibilities; clear and frequent communication in team; feedback regularly sought and accepted; team performance emphasized over individual performance; organizational reward structure appropriate for teams; and team skills training available. TeamLeadership: team leadership appropriate to task; team leader(s) serve as role models; team leader(s) technically strong; team leader(s) politically strong; and team leader(s) focused on process.
717
Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE
Proceedings of the 29th Annual Hawaii International Conference on System Sciences - 1996
Table 1 Ratings of project management and process management factors Category
Before During Change
Tools Team Leadership Customer Focus Metrics Team Background Methodologies Project Management Teamwork Behaviors Process Focus Quality Focus Change Management
2.87 2.64 2.61 2.60 2.53 2.31 2.17 1.87 1.76 1.74 1.63
3.12 3.36 3.12 2.73 2.87 2.75 3.89 2.93 3.48 2.89 3.59
they reveal if specific changes necessary for process improvement (e.g. changes in policy and/or procedure) are being recognized by the team members. Third, they reveal if possible negative side effects of change (e.g., ambiguous requirements; things taking longer than expected) are being Observed by team members. Management should be interested in these results because they provide feedback that necessary change measures are apparent and are recognized by the team members. The results also serve as indicators of potential problems when they show high awareness of negative factors such as ambiguous requirements and work taking longer to complete.
+ .25 + .72 + .51 + .13 + .34 + .44 +1.72 +1.06 +1.72 +1.15 +1.96
Table 2 Ratings of transition issues
Table 1 presents the survey results arranged from strongest to weakest category according to the first survey. The first survey measured the respondents’ perceptions of the strength of these factors as they existed in thepre-process improvement environment (the “before” column). The second column (“during”) shows the respondents’ perceptions of these same factors 6 months after the software process improvement effort was initiated by the organization. The third column shows the change from the first to the last survey. It is noteworthy that the five lowest rated categories from the first survey showed the largest improvement by the time the last survey was conducted. These categories are also heavily emphasized by the CMM.
4.2
Changes in policy and/or procedure Emphasis on use of systematic, disciplined approaches Changes in emphasis on technical review/inspections Things taking longer than expected Ambiguous requirements Different people expecting different results Project outcome goals were not clear The reorganization of task flows People having to learn new skills Changes in emphasis on quality and value A temporary drop in productivity People given new responsibilities The establishment of new work relationships The emergence of new communication patterns Not knowing who is in charge of the project Increase in cost of project Change efforts appear fast paced Working with departments in new ways Recognition of employee efforts to change General information about the progress of change
Survey number 2 -- Issues encountered in transition
This survey presents frequently-cited issues that accompany organizational change efforts. These issues were identified from a literature review [3,5,22,39,61]. Survey participants were asked to evaluate the extent to which they observed each of the following during the initial stages of the process improvement effort. A five point Likert scale was used with the following values: 1 = Never; 2 = Infrequently; 3 = Half the time; 4 = Frequently; 5 = Always. Results of this survey are presented in order of the strength of the response. The mean score for each of the items in the table is presented. This survey represents the perceptions of the respondents concerning their awareness of the following changes since the inception of the software process improvement effort. These responses reveal several things. First, they simply reveal that the respondents are cognizant of these changes in their environment. Second,
4.3
4.6 4.5 4.4 4.4 4.3 4.2 4.2 4.2 4.1 4.0 3.9 3.8 3.5 3.1 3.0 3.0 2.5 2.4 2.3 2.2
Survey number 3 -- change management strategies observed
The following change management strategies were identified from the literature review [1,22,58]. Survey participants were asked to evaluate the extent to which 718
Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE
Proceedings of the 29th Annual Hawaii International Conference on System Sciences -
5.
they observed each of the change management strategies being utilized to enhance the initial stages of the software process improvement effort. A five point Likert scale was used with the following values: 1 = Never; 2 = Infrequently; 3 = Half the time; 4 = Frequently; 5 = Always. Results of this survey are presented in order of the strength of the response. The mean score for each of the items in the table is presented. These responses show which change management strategies being employed by the organization are most visible to the respondents. Management should be interested in these responses because they can determine if change management strategies they have instituted are effective (based on visibility). Management can also see areas that are not being addressed and perhaps should be addresses, e.g. rewarding behaviors that supported the change.
Data Analysis
Results from survey number 1 @e-process improvement status) showed that the perceptions of team members’ regarding factors related to change management, quality focus, process focus, teamwork behaviors, and project management were that these areas were not well developed or clearly evident in their previous software development efforts. In addition, many individual factors of the customer focus and project management categories showed strong perceptions of weakness. These results strongly indicate that the software development efforts previously undertaken by this team could fairly be characterized as representative of level 1 (initial and ad hoc) of the CMM. It is clear that if this software development team is to successfully move to level 2 (repeatable) of the CMM, they will have to address all of these weak process management and project management factors in addition to actually developing the software. Two months after the software process improvement effort was initiated in this organization these team members were surveyed again (surveys number two and three) concerning their perceptions of the extent to which certain typical change or transitional issues were being encountered. Numerous issues were rated high in this survey including: changes in policy and/or procedure; emphasis on use of systematic, disciplined approaches; changes in emphasis on technical review/inspections; things taking longer than expected, and people having to learn new skills. Survey number three addressed team members’ perceptions of change management strategies being employed by the organization. Even the higher rated items in this survey such as communicating policy, procedures, and processes and stressing the benefits of change were not relatively high on the Likert scale. The results from surveys two and three indicated areas important to process improvement that needed immediate attention by the organization. Training workshops and seminars were provided by organizational management for of processes, requirements areas such definition management, and technical reviews and inspections. Information sessions were also organized for areas that showed weakness such as recognizing employee efforts to change and communicating status and progress regarding the software process improvement effort. Results from survey four showed that the respondents perceived previously weak process management and project management categories such as change management, quality focus, process focus, teamwork behaviors, and project management to be much stronger after the software process improvement effort had been
Table 3 Ratings of change management strategies Communicating policy, procedures, processes Engaging in ongoing project evaluation Updating the project plan as required Stressing the benefits of the change Communicating mission, values, quality Focusing on process of change Redirecting resources to bring about the change Facilitating non-hierarchical working patterns Encouraging a team effort Asking for feedback about the purpose Creating excitement about the new opportunities Cheerleading by managers for the change Recognizing a well-respected project leader Negotiating to maintain cooperation Obtaining everyone’s voluntary involvement Sticking to the schedule and timetables Building management commitment to desired results Providing education and training for those involved Rewarding behaviors that supported the change Rewarding improved performance
1996
3.3 3.2 3.1 3.0 3.0 2.9 2.8 2.6 2.5 2.5 2.5 2.4 2.4 2.3 2.3 2.2 2.2 2.2 2.1 1.8
719
Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE
Proceedings
of
the 29th Annual Hawaii International Conference on System Sciences - 1996 improvement effort to succeed. Additional training and information sessions, based on direct feedback from the software professionals, may be necessary to ensure the success of that effort. These results indicate that many organizations may need to place increased emphasis on change management strategies and a team-based, qualityfocused, process-dependent organizational culture and structure if they are to effectively undertake software process improvement initiatives. The feedback and facilitation process described in this research study can benefit software development project managers and organizational managers as they attempt to maintain continuity between the existing organizational culture, management processes, and software development software process environment and implement improvement efforts that ultimately affect all of these elements and the people involved. Additonal results will be obtained in future stages of this long-term research project.
underway for six months, Organizational management attributed this change in perception at least in part to the facilitation they provided through training and information sessions which were developed in response to the feedback obtained from surveys two and three. Since the literature shows that many software process improvement efforts are not carried through to the feedback and facilitation process completion, undertaken by this organization may provide some guidelines to other organizations expecting measurable evidence of improvement as a result of their own software process improvement efforts. 6.
Conclusions
The importance of focusing on a wide range of process management, project management, transitional change issues, and change management strategies is highlighted by this research study of the initial stages of a large-scale, long-term software process improvement effort. It seems unlikely that any organization can fully anticipate all of the attitudes and needs that a software development team has as they are required to make dramatic changes, such as those required by the CMM, in their work environment. By directly addressing these attitudes and needs, the organization featured in this study appears to have made a good start in implementing their software process improvement program. Results of this study indicate that organizational culture and subsequent change management strategies coupled with appropriate training and information sessions can have a direct effect on the rate of progress a team of software professionals makes in adopting a CMM-based software process improvement program. Further, results from this research indicate that specific, just-in-time, training and information can be developed to enhance team performance during a software process improvement effort. In sum, software process improvement efforts as modeled after the CMM often involve radical changes in traditional team and management techniques. Software process improvement entails changes in work habits, work flow patterns and structures that require people at all levels of the organization to readjust their thinking and move away from traditional modes of operation to dramatically different and new modes of operation that promise performance and quality improvement. This research shows that even established and successful software development teams may experience uncertainty and disruption when beginning a software process improvement effort and that even committed organizations may not be initially providing all the training and reinforcement necessary for the software process
References [l] Bate, R., Garcia, S., et al (1994). A systems engineering capability maturiry model, version 1.0, Software Engineering Institute, Pittsburgh, PA. [2] Bell, R. & Burnham, J. (1991). Managing change and productivity, South-Western, Cincinnati. [3] Boehm, B. (1988). “A spiral model of software development and enhancement,”IEEE Computer, May, pp. 61-72. [4] Campbell, J. & Campbell, R. (1988). Productivity in organizations, Josey-Bass, San Francisco. [5] Constantine, L.L. (1995). Constantine on peopleware. Yourdon Press, Englewood Cliffs, NJ. [6] Constantine, L.L. (1993). “Work organization: Paradigms for project managementand organization,” Communications of the ACM, Vol. 36, No. 10, October 1993, pp. 35-43. [7] Cougar, J.D. & Zawicki, R.W. (1978). “What motivates DP professionals,” Datamation, September 1978, pp, 116-123. [8] Cougar, J.D. & Zawicki, R.W. (1980). Motivation and managementof computer personnel, NY: Wiley. [9] Cougar, J.D. (1990). “Motivating analysts and programmers,” Computerworld, January 15, 1990, pp. 73-76. [lo] Curtis, B., Hefley, W.E., & Miller, S. (1995). People capability maturity model draft version 0.3, Software Engineering Institute, Pittsburgh, PA. [Ill Curtis, B. & Pat&, M. (1993). “Creating a software process improvement program,” Znformation and Sofhvre Technology, Vol. 35, No. 6, June/July 1993, pp. 381-386. [12] Daskalantonakis, M. (1994). “Achieving higher SE1 levels,” IEEE Computer, July 1994, pp. 17-24. [13] Davis, A.M. (1994). “Fifteen principles of software engineering,” IEEE Soffware, November 1994, pp. 94-96, 101. [14] Demarco, T. & Lister, T. (1987). Peopleware; Productive projects and teams. Dorsett House Publishing, NY. [15] Farwell, D.W., Kuramoto, L., Lee, D., Trauth, E.M., & Winslow, C. (1992). “A new paradigm for IS: The educational
720
Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE
Proceedings of the 29th Annual Hawaii International Conference on System Sciences implications,” Journal of Information Systems Management, Spring 1992, pp. 7-14. [16] Finkelstein, A. (1992). “A software process immaturity model,” ACM SIGSOFT, October 1992, PP.22-23. [17] Glass, R. L. (1995). So&are creativity. Prentice-Hall, Englewood Cliffs, NJ. [18] Gulden, G.K. & Reck, R.H. (1992). “Combining quality and reengineering efforts for process excellence,” Information Strategy: The Executive’s Journal, Spring 1992, pp. 10-16. 1191 Guterl, F. (1984). “Design case history: Apple’s Macintosh,” IEEE Spectrum, Vol. 21, No. 12, December, pp. 34-44. [20] Guzzo, R.A., Salas, E., et al (1995). Team efictiveness and decision making in organizations, Jossey-Bass,CA [21] Harel, D. (1992). “Biting the silver bullet: Toward a brighter future for system development,” IEEE Computer, January 1992, PP. 8-20. [22] Holtzblatt, K. & Beyer, H. (1993). “Making customercentered design work for teams,” Communicationsof the ACM, Vol. 36, No. 10, October 1993, pp. 93-103. [23] Humphrey, W. (1987). Managing for innovation: Leading technical people. Prentice-Hall, Englewood Cliffs, NJ. [24] Humphrey, W. (1989). Managing the software process. Addison-Wesley, Reading, MA. [25] Humphrey, W., Snyder, T.R., Willis, R.R. (1991). “Sollware process improvement at Hughes Aircraft,” IEEE Sojbvare, July 1991, pp. 11-23. [26] Hutchings, T., Hyde, M.G., Marca, D. & Cohen, L. (1993). “Process improvement that lasts: An integrated training and consulting method,” Communications of the ACM, Vol. 36, No. 10, October 1993, pp. 105113. [271 Hyman, R.B. (1993). “Creative chaos in high-performance teams: An experience report,” Communications of the ACM, Vol. 36, No. 10, October 1993, pp. 57-60. [28] Igbaria, M., Parasuraman, S., & Badawy, M.K. (1994). “Work experiences, job involvement, and quality of work life among information systems personnel,” MIS Quarterly, June 1994, pp. 175201. [29] Jones, D.W. (1995a). “Repeatable software development, part I: Quality Assurance,” Software Development, Vol. 3, No. 5, May 1995, pp. 59-61. [30] Jones, D.W. (1995b). “Repeatable so&ware development, part II: Configuration and requirements management,”Software Development, Vol. 3, No. 6, June 1995, pp. 55-57. [31] Jones, D.W. (1995~). “Repeatable software development, part III: Software management, Software Development, Vol. 3, No. 7, July 1995, pp. 47-51. 132) Khajenoori, S. (1994). “Process-oriented software education,” IEEE Software, November 1994, pp. 99-101. [33] Kidder, T. (1981). The Soul of a New Machine, Little, Brown, and Company: Boston, MA. [34] Kim, K.K. & Umanath, N.S. (1993). “Structure and perceived effectivenessof software developmentsubunits: A task contingency analysis,” Journal of Management Information Systems, Vol. 9, No. 3, pp. 157-181. Managing productivity in [35] Kopclman, R. (1986). organizations, McGraw-Hill, New York. [36] Kraut, R.E. & Streeter, L.A. (1995). “Coordination in
721
Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE
1996
software development,” Communications of the ACM, Vol. 38, No. 3, March 1995, pp. 69-81. [37] McIntyre, S.C. (1992). “Overcoming ad hoc development: Enabling/Disabling factors in software quality maturation,” Proceedings of the 25th Hawaii International Conference on Systems Sciences, IEEE Computer Society Press, Vol IV, January 1992, pp. 624-633. [38] McIntyre, S.C. (1994). “Organizational issues underlie software quality,” Proceedings of the 27th Annual Hawaii International Conference on System Sciences, IEEE Computer Society Press, January 1994, pp. 820-821. [39] Myers, W. (1994). “Hard data will lead managers to quality,” IEEE Sofbvare, March 1994, pp. 100-101. [40] Niederman, F., Brancheau, J.C., &Wetherbe, J.C. (1991). “Information systems management issues for the 199Os,”MIS Quarterly, December 1991, pp. 475-500. [41] Paulk, M., Curtis, Chrisis, M.B., &Weber, C.V. (1993). “Capability Maturity Model, Version 1.1 ,I’IEEE SoRware, July 1993, pp. 18-27. [42] Pierce, K.R. (1993). “Rethinking academia’s conventional wisdom,” IEEE Sofiware, March 1993, pp. 94-95, 99. [43] Pressman, R.S. (1995). “A people-centered approach to software processimprovement,” Software Process Improvement Forum, Vol. 2, No. 2, March/April 1995, pp. 8-12. [44] Rasch, R.A. & Tosi, H.L. (1992). “Factors affecting software developers’ performance: An integrated approach,” MIS Quarter&, September 1992, pp. 395-409. [45] Rettig, M. & Simons, G. (1993). “A project planning and development process for small teams,” Communications of the ACM, Vol. 36, No. 10, October 1993, pp. 45-55. [46] Royce, W.W. (1970). “Managing the development of large software systems: Concepts and techniques,” Proceedings of WESTCON, California. (Also available in Proceedings of KSE 9. Los Alamitos, California: IEEE Computer Society.) [471 Saiedian, H. &Kuzara, R. (1995). “SE1 capability maturity model’s impact on contractors,” IEEE Computer, January 1995, pp. 16-26. [48] Srinivasan, A. & Kaier, K. (1987). “Relationships between selected organizational factors and systems development,” Communications of the ACM, Vol. 30, pp. 556-562. I491 Statz, J.A. (1994). “Training effective project managers,” American Programmer, Vol. 7, No. 6. [SO]Thompson, K. & McParland, P. (1993). “Software process maturity (SPM) and the information systems developer,” Information andSo$ware Technology, Vol. 35, No. 6, June/July 1993, pp. 331-338. [Sl] Walz, D.B., Elam, J.J., & Curtis, B. (1993). “Inside a software design team: Knowledge acquisition sharing and integration,” Communications of the ACM, Vol. 36, No. 10, October 1993, pp. 63-77. [52] Werther, W., Ruth, W., & McClure, L. (1986). Productivity through people, West, St. Paul, MN. [53] Zahniser, R.A. (1993). “Design by walking around,” Communicationsof the ACM, Vol. 36, No. 10, October 1993, pp. 115-123. IS41 Zultner, R.E. (1993). “TQM for technical teams,” Communications of the ACM, Vol. 36, No. 10, October 1993, pp. 79-91.
Initial Effects of Software Process Improvement on an Experienced Software Development Team Eugene G. McGuire Computer Science and Information Systems The American University Washington, D.C. 200164116
Abstract
(as judged by customer requirements) software products. This team is now being required to implement new processes and procedures with the goal, as specified by organizational management in adopting the CMM as their model, of institutionalizing quality and reliability in their software systems. The issue being examined is the effect of the organizationally mandated software process improvement initiative on the group dynamics and team effectiveness of the existing software development team. It was anticipated by organizational management that the past success of this team would contribute to their perceptions of and reactions to the software process improvement effort. Specifically, it was expected that the software development team would prefer the previous (and proven -- in their eyes) development environment over the new (and unproven) development environment which would result from undertaking the software process improvement effort using the CMM. Organizational management hoped to facilitate this potentially difficult transition period by monitoring the perceptions and reactions of the software development team members during the initial stages of the process improvement effort and then providing training where needed. By capturing the team’s perceptions of their current strengths and their observations of change-related issues and current change management strategies employed by the organization, organizational management hoped to gain insight into the team’s attitudes (positive and negative) towards the software process improvement effort. Organizational management would then provide training in the form of workshops or seminars on issues identified as potentially troublesome or simply provide information sessions on issues that appeared to be not clearly understood by the team. For example, as expected the initial survey taken by the team showed few strengths in process focus.
Zhis paper discusses the initial stages of a long-term case study designed to examine the eflorts of an experienced software development team in moving to a more process-driven software development environment. This team has previously produced sojiware that has met functional, schedule, and cost criteria as specified by their customers. The sofiare development manager, as directed by an organizational mandate, has initiated efforts to move this team into a more structured, formalism-based development and testing environment. The Capability Maturity Model (CMM) developed by the Software Engineering Institute (SEI) is being used as the model for this effort. The group dynamic and team development aspects of this e#ort are being carefilly monitored to determine possible sources of resistance to change and to develop intervention “just-in-time” training sessions that can address identif?ed problem areas, particularly those that may directly affect productivity, quality, andschedule. This paper discussesinitialfindings in this area and addresses them within the CMM framework.
1.
Introduction
Software development teams, because of their often mission-critical status, are increasingly in the forefront of change processes in an organization. Accompanying the increased reliance on effective software teams in general is a growing emphasis on quality products and services that are often produced through rigorous accountability procedures designed to standardize process and quantify data measurement [13,18,21]. This research study examines the effects of a software process improvement effort on an experienced software development team that has previously produced successful
1060-3425/96 $5.0001996IEEE
713
Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE
Proceedings of the 29th Annual Hawaii International
Literature Review
2.1
Software Process Improvement
1996
under the CMM framework), managers use metrics to quantitatively measure the quality of a software product. Schedules and budgets are based on historical performance and are realistic; the predicted results for product cost, schedule, functionality, and quality are routinely achieved. Based on the predictability of the project results, reliable management decisions can be made about tradeoffs among these outcomes. Models such as the CMM are designed to contribute toward lower costs and reduced intervals because they specify high quality processes. These processes are capable of predictable results (producing high quality software products) which in turn help with the planning and management of projects.
Organizational management was then able to justify and able to institute a series of workshops designed to explain in detail how specific software processes could be improved in that environment. This went far beyond the initial introduction to CMM training that had been originally planned for the team. One example of improving information flow resulted from the team’s perception that employee efforts to change were not well recognized by the organization (see Table 2). As a result of this feedback, organizational management held an information session with the team that focused on rewarding positive efforts and recognizing role models. Some of the questions this research attempts to address include examining whether the volatility of internal software engineering efforts can be affected by: team structure and norms of operation, integration of process into team responsibilities, and control activities coordination with management in team development and responsibility. This research can potentially benefit organizational managers and software development project managers as they attempt to facilitate a smooth transition from a nonprocess driven software development environment to a process-focused environment under the CMM.
2.
Conference on System Sciences -
2.2
Managing People and Teams
Management can also more clearly define relationships concerning roles and responsibilities between software developers and users. This approach recognizes the importance of communication between customer and software developer as well as between professionals with different skills. For example, it is inevitable that teams of IS and functional area professionals formed to create complex information systems bring distinctive varieties of human behavior and learning which may deeply affect the resulting system to any complex development effort. This diversity in style occurs despite structure imposed by task requirements, group behavior norms, and extemallydeveloped guidelines such as the CMM. Nevertheless, it is expected that by establishing and enforcing formal procedures for communication and collaboration as indicated under the CMM, team performance can be enhanced and problems diminished. Another aim of this approach is to move the emphasis of the development from being an artistic expression of individuals working mostly alone to one of team-based engineering. This is not an easy task as some research [7,8,9] shows that the single most important motivator for data processing professionals is the creativity of the job itself. In addition, self-managed teams appear to be evolving at a rapid pace as a result of TQM and reengineering initiatives in many organizations. The literature shows many studies which demonstrate that extraordinary productivity results when people are placed in selfmanaged teams in highly constrained environments. There is a substantial literature relating tales of “heroic” operations working against impossible deadlines with minimal resources, thereby becoming extraordinarily motivated and sometimes flouting the expectations of the mature parent organization. Some of these projects were poorly structured but, nevertheless, succeeded as a result of the personalities of the leaders [19,33].
Software process improvement arose out of the quality principles of Deming, Juran, and Crosby in the mid-1980s in response to the belief that better management of software development processes would lead to improved software reliability and quality. One of the better recognized models of software process maturity is the Capability Maturity Model (CMM) developed by the Software Engineering Institute (SEI) at Carnegie Mellon University in Pittsburgh [ 11,411. Software process improvement as typified by the CMM provides guidelines for an organization to employ a structured approach to strategically improve all aspects of a software development operation and dramatically changes or extends concepts previously employed under recognized systems development models such as the waterfall and spiral approach [3,46]. Organizations previously would often attempt to improve performance in an ad-hoc manner by using technology driven approaches instead of addressing root problems. For example, organizations would often assign more staff or more tools to an already late and over-budget development project rather than addressing the reasons why the project had problems. In contrast, in a mature organization (as measured
714
Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE
Proceedings of the 29th Annual Hawaii International Conferenceon SystemSciences- 1996 members are not well defined. Project and design information is not stored or communicated effectively. The organization does not consistently apply good engineering management to the software development process. Descriptors such as “ad-hoc” and “chaotic” have been frequently applied to this level. Early assessments done by the SE1 found that more than 70% of assessed organizations were at Level 1. Level 2 -- Repeatable: The organization has achieved a stable process with a repeatable management control level by initiating rigorous project management of commitments, costs, schedules, and changes. The organization uses standard practices for managing development activities including: staff estimating; cost estimating; scheduling; change and version control on and design data; and requirements, specifications, managerial reviews. The managers are trained in these management practices and software engineers are trained on the use of tools and procedures. The organization is good at doing similar types of work but is challenged by new types of work because it lacks an orderly framework for improvement. Level 3 -- Defined: The organization has defined the process as a basis for consistent implementation and better understanding. Goals, objectives and strategies are developed and promulgated. The software development organization is multi-disciplinary and a Software Engineering Process Group (SEPG) is established to lead process improvement. At this point, the risk of introducing advanced technology is greatly reduced. Level 4 -- Managed: The organization has initiated comprehensive process measurements and analysis. This is when the most significant quality improvements begin. The senior managers are involved in the product and process management. The performance of the design process is monitored and measured; data gathering is repeatable and auditable. The organization typically bases its operating decisions on quantitative process data, and conducts extensive analysis of the data gathered during reviews and tests. Early assessments conducted by SE1 found no organizations operating at this level and recent assessments show only a few at this level. Level 5 -- Optimized: The organization now has a foundation for continuously improving and optimizing the process. The organization conducts analyses of the metrics data gathered during the design process and use these for planning improvements. They focus on improving and optimizing the process and products by establishing improvement goals for quality, productivity, and cost. Early assessments conducted by SE1 found no organizations operating at this level and recent assessments show only a very few at this level. Two variations on the CMM are the Systems
Much professional literature exists that attempts to predict the future of the information systems discipline, especially in relationship to managing people and teams involved in complex projects [5,14,17,23]. A closely related stream of research literature focuses on the dominant information technology issues of the 1990s [40] and work performance issues of software developers as they attempt to cope with complex systems development projects [28,44]. Common themes include: systems developers rapidly approaching their limits in terms of handling complexity, organizational managers increasingly disappointed by the lack of contribution of information systems to profits, the chaos of personal computing, and the overall failure of existing approaches for developing successful systems. Indeed, as complexity grows, practitioners and academics alike are examining and revising educational requirements for software engineers [l&32,42].
2.3
Characteristics of Team-Based Process Maturity
Part of the reason for the recent attention to quality in software development is that empirical research consistently shows that many software development projects suffer from a lack of proven and well established methods [16]. Models such as the CMM are only now advancing beyond the embryonic stage and being accepted by a wide variety of organizations [25]. They still have not yet undergone the rigorous test of years of industrial application. In fact, many organizations that begin assessments, do not follow through in improving their development processes [39]. Many organizations, nevertheless, employ talented software engineers and managers and produce successful software products even though they are not operating with a strong process focus [12,25]. In such organizations, it is appropriate to examine if team and management issues arise as those software development groups move towards a more structured, process-oriented environment. The CMM requires that organizational management, users, customers, and software development professionals communicate and collaborate frequently and effectively to achieve quality and repeatable processes. In fact, team training is specifically required to fully satisfy the requirements of some key process areas in the CMM. The CMM breaks down the software engineering capabilities of organizations into 5 maturity levels from Level 1 (Initial) to Level 5 (Optimizing). Generally, the levels can be characterized and distinguished in terms of team and management processes as: Level 1 -- Initial: There are no well-defined procedures and management. Roles of the design team
715
Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE
Proceedings of the 29th Annual Hawaii International
related to the existing software development environment. This survey was conducted to determine the perceived current (or pre-process improvement) status of these factors. Seventy-nine factors were identified from a literature review and categorized under the categories of: Teamwork Behaviors, Team Team Background, Leadership, Change Management, Quality Focus, Process Project Management, Focus, Customer Focus, Methodologies, Metrics, and Tools. These factors were defined and discussed with team members prior to the survey. Survey participants were asked to rate the current status of the 79 factors on a Likert scale. Second, team members were surveyed two months later on their perceptions concerning the extent to which various change-related issues (again identified through a literature review) were encountered during the initial stages of the process improvement effort as the team was making the transition from an old to a new way of operating. The Likert scale survey measured the team members’ perceptions of the frequency with which 20 typical organizational change issues had arisen. Third, at the same time, they were asked to evaluate the extent to which they observed various change management strategies being employed by the organization and their managers during the transition period from an “ad-hoc” development environment to a more structured development environment. This survey of 20 typical change management strategies was also done with a Likert scale. Results from these three initial surveys were used by organizational management and software development management to develop and provide training and information sessions for team members and management in the software development environment. Fourth, team members were surveyed six months after the software process improvement effort was initiated (four months after they identified change-related issues and received training related to these issues) on the same set of 79 factors related to the software development environment that they addressed in the first survey. This survey was conducted to determine the perceived status of these factors after the initial stage of the process improvement effort was underway. A comparison between the responses of the team’from the first to the last survey in this study reveals their perceptions of the differences in the status or relative maturity of the factors in the survey after the software process was underway. This research employs Likert-scale surveys and is intended to capture the perceptions of the respondents. It is recognized that perceptions can be formed by many factors and that each individual respondent brings their own experience and interpretation to each item in the survey. These are well-recognized limitations of Likert-
Engineering Capability Maturity Model (SE-CMM), Version 1.0 (December 1994) [l] and the People Management Capability Maturity Model (P-CMM), Version 0.3 (April, 1995) [lo]. These variations of the original CMM seek to focus the organization’s resources on the whole systems engineering process and the human resources aspects of the processes respectively. By extending the initial CMM framework to cover systems and engineering and human resource factors the SE1 is attempting to address problems that fall outside the software domain. Both of these new models, particularly the I’-CMM include substantive discussion of how people, technical and end-user, are instrumental in the production of quality software/systems. Most improvement programs to date have emphasized process or technology and not people. People management practices in many organizations, despite a significant amount of literature addressing people-related issues [34,36,43,44,48,50], do not address people issues in a systematic and structured manner. In addition, most managers are untrained or inadequately trained in implementing corrective solutions once people problems are identified [45,49,53]. Also, organizational factors, while widely cited [6,37,38,47] are often not systematically analyzed for their effect on process improvement efforts. The characteristics of the SE1 levels show that information systems professionals who work on software engineering projects must be capable of being highly productive in complex, team-oriented environments with strong emphases on process control and overall quality [51,54]. These requirements are critical but may not always be present in current information systems professionals. The lack of appropriate team and process control skills can greatly contribute to internal organizational volatility. This research study attempts to address these issues.
3.
Conference on System Sciences - 1996
Research Methodology
Sixty four software development professional comprised the team that is the subject of this study. The team included project managers, analysts, programmers, testers, configuration management specialists, and documentation specialists. They worked on separate, but related, modules of the same project. Some of their interactions were within the module groups and some were within the larger team. Team members had on average approximately two years of experience woking in this team. This research study was conducted by first surveying the members of the software development team on their perceptions of the strength of a wide variety of factors
716
Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE
Proceedings of the 29th Annual Hawaii International Conference on System Sciences - 1996 scale surveys and therefore care should be taken in generalizing these results to other teams and/or organizations. The advantages of Liiert-scale surveys, however, are that they are a means for quick and efficient data collection and can indicate trends in the thinking of the respondents as a whole. Additional insights and richer detail can be obtained through other research methods such as interviews and will be part of future stages of this long-term study. The value of this particular research study and the methodology is in the process employed as opposed to the specific results of the surveys. The results obtained from surveys such as these will obviously vary depending on the organizational and software development environment. The specific results from the surveys in this study, therefore, may not be generalizable to other organizations. However, the process of utilizing Likert-type surveys to take the temperature of the team regarding project management and process management issues is valuable if the organization is committed to responding to this feedback to facilitate a smooth transition to a processdriven software development environment.
4.
Data Results
4.1
Survey number 1 & 4 -- Pre-process improvement status and in-process improvement status
Change Management: organization communicates goals; organization communicates values; negotiation used to obtain agreement and cooperation; behavior supporting change rewarded; resources provided/redirected to support change; and increased participation and decision making encouraged. Quality Focus: pre-delivery defect reports; test status reports; project post-mortems; quality plan; defect distribution through code; quality assurance function; root analysis; cause quality assurance reports; review/inspection defect removal efficiency; and errorprone module analysis. ProcessFocus: software process definitions; software process monitoring; software process assessment; software process integrated with organizational objectives; and software process practices reflect state-of-the-art. Customer Focus: customer support; customer defect reports; customer requests for enhancement; customer satisfaction surveys; customer involvement during acceptance testing; customer acceptance testing; customer satisfaction with documentation; and customer involvement during customer documentation. Project Management:project/development plan; actual vs. estimated schedule; milestone and progress reviews; software estimation methods and tools; project management audit; schedule estimation process effectiveness; review/inspection scheduling; effort estimation process effectiveness; and product plan. Methodologies: requirements review; functional (high level) design; system test plan; testing function; system testing; software development processes; system test plan review; data dictionary; software architecture; reuse; reengineering; and prototyping. Metrics: defect quantities; defect severities; response time; throughput; life cycle productivity measurements; and reused code size. Tools: program debugging tools; support software effectiveness; internal documentation tool effectiveness; and source code project library. A five-point Likert scale was used with the following values: 1 = None; 2 = Weak; 3 = Average; 4 = Strong; 5 = Excellent. Results from the first and last surveys are summarized in the following table. The mean score for all the factors in each category has been averaged for pre-process improvement (before --beginning of research study) and in-process improvement (during -after 6 months). The mean scores for each of the items in the 11 categories in Table 1 were captured in the data collection phase of this study and can be used by organizational management and software development project management to identify specific weaknesses in each of these categories. Those details are not presented here.
At the beginning of this research study (pre-process improvement) and after six months of process improvement efforts survey participants were asked the following question: “In your opinion and based on your experience in your current software development team, how ‘would you rate the following in your team:” Respondents were presented with 11 categories of project and process related factors. There were 79 total factors. The categories and factors in those categories are: Team Background: project team experience; programming language experience; application experience; project management experience; software engineering training for staff; review and inspection experience; and training plan. Teamwork Behaviors: clearly defined roles and responsibilities; clear and frequent communication in team; feedback regularly sought and accepted; team performance emphasized over individual performance; organizational reward structure appropriate for teams; and team skills training available. TeamLeadership: team leadership appropriate to task; team leader(s) serve as role models; team leader(s) technically strong; team leader(s) politically strong; and team leader(s) focused on process.
717
Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE
Proceedings of the 29th Annual Hawaii International Conference on System Sciences - 1996
Table 1 Ratings of project management and process management factors Category
Before During Change
Tools Team Leadership Customer Focus Metrics Team Background Methodologies Project Management Teamwork Behaviors Process Focus Quality Focus Change Management
2.87 2.64 2.61 2.60 2.53 2.31 2.17 1.87 1.76 1.74 1.63
3.12 3.36 3.12 2.73 2.87 2.75 3.89 2.93 3.48 2.89 3.59
they reveal if specific changes necessary for process improvement (e.g. changes in policy and/or procedure) are being recognized by the team members. Third, they reveal if possible negative side effects of change (e.g., ambiguous requirements; things taking longer than expected) are being Observed by team members. Management should be interested in these results because they provide feedback that necessary change measures are apparent and are recognized by the team members. The results also serve as indicators of potential problems when they show high awareness of negative factors such as ambiguous requirements and work taking longer to complete.
+ .25 + .72 + .51 + .13 + .34 + .44 +1.72 +1.06 +1.72 +1.15 +1.96
Table 2 Ratings of transition issues
Table 1 presents the survey results arranged from strongest to weakest category according to the first survey. The first survey measured the respondents’ perceptions of the strength of these factors as they existed in thepre-process improvement environment (the “before” column). The second column (“during”) shows the respondents’ perceptions of these same factors 6 months after the software process improvement effort was initiated by the organization. The third column shows the change from the first to the last survey. It is noteworthy that the five lowest rated categories from the first survey showed the largest improvement by the time the last survey was conducted. These categories are also heavily emphasized by the CMM.
4.2
Changes in policy and/or procedure Emphasis on use of systematic, disciplined approaches Changes in emphasis on technical review/inspections Things taking longer than expected Ambiguous requirements Different people expecting different results Project outcome goals were not clear The reorganization of task flows People having to learn new skills Changes in emphasis on quality and value A temporary drop in productivity People given new responsibilities The establishment of new work relationships The emergence of new communication patterns Not knowing who is in charge of the project Increase in cost of project Change efforts appear fast paced Working with departments in new ways Recognition of employee efforts to change General information about the progress of change
Survey number 2 -- Issues encountered in transition
This survey presents frequently-cited issues that accompany organizational change efforts. These issues were identified from a literature review [3,5,22,39,61]. Survey participants were asked to evaluate the extent to which they observed each of the following during the initial stages of the process improvement effort. A five point Likert scale was used with the following values: 1 = Never; 2 = Infrequently; 3 = Half the time; 4 = Frequently; 5 = Always. Results of this survey are presented in order of the strength of the response. The mean score for each of the items in the table is presented. This survey represents the perceptions of the respondents concerning their awareness of the following changes since the inception of the software process improvement effort. These responses reveal several things. First, they simply reveal that the respondents are cognizant of these changes in their environment. Second,
4.3
4.6 4.5 4.4 4.4 4.3 4.2 4.2 4.2 4.1 4.0 3.9 3.8 3.5 3.1 3.0 3.0 2.5 2.4 2.3 2.2
Survey number 3 -- change management strategies observed
The following change management strategies were identified from the literature review [1,22,58]. Survey participants were asked to evaluate the extent to which 718
Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE
Proceedings of the 29th Annual Hawaii International Conference on System Sciences -
5.
they observed each of the change management strategies being utilized to enhance the initial stages of the software process improvement effort. A five point Likert scale was used with the following values: 1 = Never; 2 = Infrequently; 3 = Half the time; 4 = Frequently; 5 = Always. Results of this survey are presented in order of the strength of the response. The mean score for each of the items in the table is presented. These responses show which change management strategies being employed by the organization are most visible to the respondents. Management should be interested in these responses because they can determine if change management strategies they have instituted are effective (based on visibility). Management can also see areas that are not being addressed and perhaps should be addresses, e.g. rewarding behaviors that supported the change.
Data Analysis
Results from survey number 1 @e-process improvement status) showed that the perceptions of team members’ regarding factors related to change management, quality focus, process focus, teamwork behaviors, and project management were that these areas were not well developed or clearly evident in their previous software development efforts. In addition, many individual factors of the customer focus and project management categories showed strong perceptions of weakness. These results strongly indicate that the software development efforts previously undertaken by this team could fairly be characterized as representative of level 1 (initial and ad hoc) of the CMM. It is clear that if this software development team is to successfully move to level 2 (repeatable) of the CMM, they will have to address all of these weak process management and project management factors in addition to actually developing the software. Two months after the software process improvement effort was initiated in this organization these team members were surveyed again (surveys number two and three) concerning their perceptions of the extent to which certain typical change or transitional issues were being encountered. Numerous issues were rated high in this survey including: changes in policy and/or procedure; emphasis on use of systematic, disciplined approaches; changes in emphasis on technical review/inspections; things taking longer than expected, and people having to learn new skills. Survey number three addressed team members’ perceptions of change management strategies being employed by the organization. Even the higher rated items in this survey such as communicating policy, procedures, and processes and stressing the benefits of change were not relatively high on the Likert scale. The results from surveys two and three indicated areas important to process improvement that needed immediate attention by the organization. Training workshops and seminars were provided by organizational management for of processes, requirements areas such definition management, and technical reviews and inspections. Information sessions were also organized for areas that showed weakness such as recognizing employee efforts to change and communicating status and progress regarding the software process improvement effort. Results from survey four showed that the respondents perceived previously weak process management and project management categories such as change management, quality focus, process focus, teamwork behaviors, and project management to be much stronger after the software process improvement effort had been
Table 3 Ratings of change management strategies Communicating policy, procedures, processes Engaging in ongoing project evaluation Updating the project plan as required Stressing the benefits of the change Communicating mission, values, quality Focusing on process of change Redirecting resources to bring about the change Facilitating non-hierarchical working patterns Encouraging a team effort Asking for feedback about the purpose Creating excitement about the new opportunities Cheerleading by managers for the change Recognizing a well-respected project leader Negotiating to maintain cooperation Obtaining everyone’s voluntary involvement Sticking to the schedule and timetables Building management commitment to desired results Providing education and training for those involved Rewarding behaviors that supported the change Rewarding improved performance
1996
3.3 3.2 3.1 3.0 3.0 2.9 2.8 2.6 2.5 2.5 2.5 2.4 2.4 2.3 2.3 2.2 2.2 2.2 2.1 1.8
719
Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE
Proceedings
of
the 29th Annual Hawaii International Conference on System Sciences - 1996 improvement effort to succeed. Additional training and information sessions, based on direct feedback from the software professionals, may be necessary to ensure the success of that effort. These results indicate that many organizations may need to place increased emphasis on change management strategies and a team-based, qualityfocused, process-dependent organizational culture and structure if they are to effectively undertake software process improvement initiatives. The feedback and facilitation process described in this research study can benefit software development project managers and organizational managers as they attempt to maintain continuity between the existing organizational culture, management processes, and software development software process environment and implement improvement efforts that ultimately affect all of these elements and the people involved. Additonal results will be obtained in future stages of this long-term research project.
underway for six months, Organizational management attributed this change in perception at least in part to the facilitation they provided through training and information sessions which were developed in response to the feedback obtained from surveys two and three. Since the literature shows that many software process improvement efforts are not carried through to the feedback and facilitation process completion, undertaken by this organization may provide some guidelines to other organizations expecting measurable evidence of improvement as a result of their own software process improvement efforts. 6.
Conclusions
The importance of focusing on a wide range of process management, project management, transitional change issues, and change management strategies is highlighted by this research study of the initial stages of a large-scale, long-term software process improvement effort. It seems unlikely that any organization can fully anticipate all of the attitudes and needs that a software development team has as they are required to make dramatic changes, such as those required by the CMM, in their work environment. By directly addressing these attitudes and needs, the organization featured in this study appears to have made a good start in implementing their software process improvement program. Results of this study indicate that organizational culture and subsequent change management strategies coupled with appropriate training and information sessions can have a direct effect on the rate of progress a team of software professionals makes in adopting a CMM-based software process improvement program. Further, results from this research indicate that specific, just-in-time, training and information can be developed to enhance team performance during a software process improvement effort. In sum, software process improvement efforts as modeled after the CMM often involve radical changes in traditional team and management techniques. Software process improvement entails changes in work habits, work flow patterns and structures that require people at all levels of the organization to readjust their thinking and move away from traditional modes of operation to dramatically different and new modes of operation that promise performance and quality improvement. This research shows that even established and successful software development teams may experience uncertainty and disruption when beginning a software process improvement effort and that even committed organizations may not be initially providing all the training and reinforcement necessary for the software process
References [l] Bate, R., Garcia, S., et al (1994). A systems engineering capability maturiry model, version 1.0, Software Engineering Institute, Pittsburgh, PA. [2] Bell, R. & Burnham, J. (1991). Managing change and productivity, South-Western, Cincinnati. [3] Boehm, B. (1988). “A spiral model of software development and enhancement,”IEEE Computer, May, pp. 61-72. [4] Campbell, J. & Campbell, R. (1988). Productivity in organizations, Josey-Bass, San Francisco. [5] Constantine, L.L. (1995). Constantine on peopleware. Yourdon Press, Englewood Cliffs, NJ. [6] Constantine, L.L. (1993). “Work organization: Paradigms for project managementand organization,” Communications of the ACM, Vol. 36, No. 10, October 1993, pp. 35-43. [7] Cougar, J.D. & Zawicki, R.W. (1978). “What motivates DP professionals,” Datamation, September 1978, pp, 116-123. [8] Cougar, J.D. & Zawicki, R.W. (1980). Motivation and managementof computer personnel, NY: Wiley. [9] Cougar, J.D. (1990). “Motivating analysts and programmers,” Computerworld, January 15, 1990, pp. 73-76. [lo] Curtis, B., Hefley, W.E., & Miller, S. (1995). People capability maturity model draft version 0.3, Software Engineering Institute, Pittsburgh, PA. [Ill Curtis, B. & Pat&, M. (1993). “Creating a software process improvement program,” Znformation and Sofhvre Technology, Vol. 35, No. 6, June/July 1993, pp. 381-386. [12] Daskalantonakis, M. (1994). “Achieving higher SE1 levels,” IEEE Computer, July 1994, pp. 17-24. [13] Davis, A.M. (1994). “Fifteen principles of software engineering,” IEEE Soffware, November 1994, pp. 94-96, 101. [14] Demarco, T. & Lister, T. (1987). Peopleware; Productive projects and teams. Dorsett House Publishing, NY. [15] Farwell, D.W., Kuramoto, L., Lee, D., Trauth, E.M., & Winslow, C. (1992). “A new paradigm for IS: The educational
720
Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE
Proceedings of the 29th Annual Hawaii International Conference on System Sciences implications,” Journal of Information Systems Management, Spring 1992, pp. 7-14. [16] Finkelstein, A. (1992). “A software process immaturity model,” ACM SIGSOFT, October 1992, PP.22-23. [17] Glass, R. L. (1995). So&are creativity. Prentice-Hall, Englewood Cliffs, NJ. [18] Gulden, G.K. & Reck, R.H. (1992). “Combining quality and reengineering efforts for process excellence,” Information Strategy: The Executive’s Journal, Spring 1992, pp. 10-16. 1191 Guterl, F. (1984). “Design case history: Apple’s Macintosh,” IEEE Spectrum, Vol. 21, No. 12, December, pp. 34-44. [20] Guzzo, R.A., Salas, E., et al (1995). Team efictiveness and decision making in organizations, Jossey-Bass,CA [21] Harel, D. (1992). “Biting the silver bullet: Toward a brighter future for system development,” IEEE Computer, January 1992, PP. 8-20. [22] Holtzblatt, K. & Beyer, H. (1993). “Making customercentered design work for teams,” Communicationsof the ACM, Vol. 36, No. 10, October 1993, pp. 93-103. [23] Humphrey, W. (1987). Managing for innovation: Leading technical people. Prentice-Hall, Englewood Cliffs, NJ. [24] Humphrey, W. (1989). Managing the software process. Addison-Wesley, Reading, MA. [25] Humphrey, W., Snyder, T.R., Willis, R.R. (1991). “Sollware process improvement at Hughes Aircraft,” IEEE Sojbvare, July 1991, pp. 11-23. [26] Hutchings, T., Hyde, M.G., Marca, D. & Cohen, L. (1993). “Process improvement that lasts: An integrated training and consulting method,” Communications of the ACM, Vol. 36, No. 10, October 1993, pp. 105113. [271 Hyman, R.B. (1993). “Creative chaos in high-performance teams: An experience report,” Communications of the ACM, Vol. 36, No. 10, October 1993, pp. 57-60. [28] Igbaria, M., Parasuraman, S., & Badawy, M.K. (1994). “Work experiences, job involvement, and quality of work life among information systems personnel,” MIS Quarterly, June 1994, pp. 175201. [29] Jones, D.W. (1995a). “Repeatable software development, part I: Quality Assurance,” Software Development, Vol. 3, No. 5, May 1995, pp. 59-61. [30] Jones, D.W. (1995b). “Repeatable so&ware development, part II: Configuration and requirements management,”Software Development, Vol. 3, No. 6, June 1995, pp. 55-57. [31] Jones, D.W. (1995~). “Repeatable software development, part III: Software management, Software Development, Vol. 3, No. 7, July 1995, pp. 47-51. 132) Khajenoori, S. (1994). “Process-oriented software education,” IEEE Software, November 1994, pp. 99-101. [33] Kidder, T. (1981). The Soul of a New Machine, Little, Brown, and Company: Boston, MA. [34] Kim, K.K. & Umanath, N.S. (1993). “Structure and perceived effectivenessof software developmentsubunits: A task contingency analysis,” Journal of Management Information Systems, Vol. 9, No. 3, pp. 157-181. Managing productivity in [35] Kopclman, R. (1986). organizations, McGraw-Hill, New York. [36] Kraut, R.E. & Streeter, L.A. (1995). “Coordination in
721
Proceedings of the 1996 Hawaii International Conference on System Sciences (HICSS-29) 1060-3425/96 $10.00 © 1996 IEEE
1996
software development,” Communications of the ACM, Vol. 38, No. 3, March 1995, pp. 69-81. [37] McIntyre, S.C. (1992). “Overcoming ad hoc development: Enabling/Disabling factors in software quality maturation,” Proceedings of the 25th Hawaii International Conference on Systems Sciences, IEEE Computer Society Press, Vol IV, January 1992, pp. 624-633. [38] McIntyre, S.C. (1994). “Organizational issues underlie software quality,” Proceedings of the 27th Annual Hawaii International Conference on System Sciences, IEEE Computer Society Press, January 1994, pp. 820-821. [39] Myers, W. (1994). “Hard data will lead managers to quality,” IEEE Sofbvare, March 1994, pp. 100-101. [40] Niederman, F., Brancheau, J.C., &Wetherbe, J.C. (1991). “Information systems management issues for the 199Os,”MIS Quarterly, December 1991, pp. 475-500. [41] Paulk, M., Curtis, Chrisis, M.B., &Weber, C.V. (1993). “Capability Maturity Model, Version 1.1 ,I’IEEE SoRware, July 1993, pp. 18-27. [42] Pierce, K.R. (1993). “Rethinking academia’s conventional wisdom,” IEEE Sofiware, March 1993, pp. 94-95, 99. [43] Pressman, R.S. (1995). “A people-centered approach to software processimprovement,” Software Process Improvement Forum, Vol. 2, No. 2, March/April 1995, pp. 8-12. [44] Rasch, R.A. & Tosi, H.L. (1992). “Factors affecting software developers’ performance: An integrated approach,” MIS Quarter&, September 1992, pp. 395-409. [45] Rettig, M. & Simons, G. (1993). “A project planning and development process for small teams,” Communications of the ACM, Vol. 36, No. 10, October 1993, pp. 45-55. [46] Royce, W.W. (1970). “Managing the development of large software systems: Concepts and techniques,” Proceedings of WESTCON, California. (Also available in Proceedings of KSE 9. Los Alamitos, California: IEEE Computer Society.) [471 Saiedian, H. &Kuzara, R. (1995). “SE1 capability maturity model’s impact on contractors,” IEEE Computer, January 1995, pp. 16-26. [48] Srinivasan, A. & Kaier, K. (1987). “Relationships between selected organizational factors and systems development,” Communications of the ACM, Vol. 30, pp. 556-562. I491 Statz, J.A. (1994). “Training effective project managers,” American Programmer, Vol. 7, No. 6. [SO]Thompson, K. & McParland, P. (1993). “Software process maturity (SPM) and the information systems developer,” Information andSo$ware Technology, Vol. 35, No. 6, June/July 1993, pp. 331-338. [Sl] Walz, D.B., Elam, J.J., & Curtis, B. (1993). “Inside a software design team: Knowledge acquisition sharing and integration,” Communications of the ACM, Vol. 36, No. 10, October 1993, pp. 63-77. [52] Werther, W., Ruth, W., & McClure, L. (1986). Productivity through people, West, St. Paul, MN. [53] Zahniser, R.A. (1993). “Design by walking around,” Communicationsof the ACM, Vol. 36, No. 10, October 1993, pp. 115-123. IS41 Zultner, R.E. (1993). “TQM for technical teams,” Communications of the ACM, Vol. 36, No. 10, October 1993, pp. 79-91.
