Iterative and incremental development

Iterative and Incremental development is a software development process, as opposed to more traditional waterfall model. It is especially known as a part of Extreme Programming.

The basic idea behind iterative enhancement is to develop a software system incrementally, allowing the developer to take advantage of what was being learned during the development of earlier, incremental, deliverable versions of the system. Learning comes from both the development and use of the system, where possible. Key steps in the process were to start with a simple implementation of a subset of the software requirements and iteratively enhance the evolving sequence of versions until the full system is implemented. At each iteration, design modifications are made along with addition new functional capabilities.

The Procedure itself consists of the Initialization step, the Iteration step, and the Project Control List. The initialization step creates a base version of the system. The goal for this initial implementation is to create a product to which the user can react. It should offer a sampling of the key aspects of the problem and provide a solution that is simple enough to understand and implement easily. To guide the iteration process, a project control list is created that contains a record of all tasks that need to be performed. It includes such items as new features to be implemented and areas of redesign of the exiting solution. The control list is constantly being revised as a result of the analysis phase.

The iteration step involves the redesign and implementation of a task from project control list, and the analysis of the current version of the system. The goal for the design and implementation of any iteration is to be simple, straightforward, and modular, supporting redesign at that stage or as a task added to the project control list. The code represents the major source of documentation of the system. The analysis of an iteration is based upon user feedback and the program analysis facilities available. It involves analysis of the structure, modularity, usability, reliability, efficiency, and achievement of goals. The project control list is modified in light of the analysis results.

Guidelines that drive the implementation and analysis include:

Any difficulty in design, coding and testing a modification should signal the need for redesign or re-coding.
Modifications should fit easily into isolated and easy-to-find- modules. If they do not, some redesign is needed.
Modifications to tables should be especially easy to make. If any table modification is not quickly and easily done, redesign is indicated.
Modifications should become easier to make as the iterations progress. If they are not, there is a basic problem such as a design flaw or a proliferation of patches.
Patches should normally be allowed to exist for only one or two iterations. Patches may be necessary to avoid redesigning during an implementation phase.
The existing implementation should be analyzed frequently to determine how well it measures up to project goals.
Program analysis facilities should be used whenever available to aid in the analysis of partial implementations
User reaction should be solicited and analyzed for indications of deficiencies in the current implementation.

Iterative Enhancement was successfully applied to the development of an extendable family of compilers for a family of programming languages on a variety of hardware architectures. A set of 17 versions of the system was developed at one site generating 17 thousand source lines of high level language (6500 lines of executable code). The system was further developed at two different sites, leading to two different versions of the base language: one version essentially focused on mathematical applications, adding real numbers and various mathematical functions, and the other adding compiler writing capabilities. Each iteration was analyzed from the user's point of view (the language capabilities were determined in part by the user's needs) and the developer's point of view (the compiler design evolved to be more easily modified for characteristics like adding new data types). Measurement such as coupling and modularization were tracked over multiple versions.

Using analysis and measurement as drivers of the enhancement process is one major difference between iterative enhancement and the current agile software development. It provides support for determining the effectiveness of the processes and the quality of product. It allows one to study, and therefore improve and tailor, the processes for the particular environment. This measurement and analysis activity can be added to existing agile development methods.

In fact, the context of multiple iterations provides advantages in the use of measurement. Measures are sometimes difficult to understand in the absolute but the relative changes in measures over the evolution of the system can be very informative as they provide a basis for comparison. For example, a vector of measures, m1, m 2, ... mn, can be defined to characterize various aspects of the product at some point in time, e.g., effort to date, changes, defects, logical, physical, and dynamic attributes, environmental considerations. Thus an observer can tell how product characteristics like size, complexity, coupling, and cohesion are increasing or decreasing over time. One can monitor the relative change of the various aspects of the product or can provide bounds for the measures to signal potential problems and anomalies.

History

For the June 2003 IEEE Computer issue dedicated to agile methods (edited by A. Cockburn and L. Williams), Vic Basili and CraigLarman are writing a short 1-2 page history of iterative/incremental lifecycle processes.

1970: Royce, W.W., Managing the Development of Large-Scale Software: Concepts and Techniques Proceedings, Wescon, August 1970 (also reprinted in Proceedings, ICSE9), which includes a "build it twice" prototyping step -- entered by Barry Boehm

1971: Mills, H., Top-down programming in large systems Debugging Techniques in Large Systems, R. Rustin, ed., Englewood Cliffs, N.J., Prentice-Hall, 1971. (Frederick Brooks mentions this in NoSilverBullet: "Some years ago Harlan Mills proposed that any software system should be grown by incremental development.") - entered by Christian Ohman

1973: Mills, H., On the Development of Large, Reliable Programs IEEE Symp. Comp. SW Reliability. Notes: I have heard this paper has relevance to iterative, but havent' read it yet. - CraigLarman

1975: Williams, R.D., Managing the Development of Reliable Software Proceedings, 1975 International Conference on Reliable Software, ACM/IEEE, April 1975, pp.3-8.

Discusses the use of incremental development on the $100M TRW/Army Site Defense software project for ballistic missile defense. The project began in February 1972 and developed the software in 5 loops or increments of functional capability. Loop 1 just did tracking of a single object; Loop 5 covered the full mission. -- entered by BarryBoehm

1975: Brooks, F., The Mythical Man-Month

"Plan to throw one away; you will, anyhow." - entered by PhilippeKruchten

Please note that Brooks writes in The Mythical Man-Month after 20 years: "This I now perceive to be wrong, not because it is too radical, but because it is too simplistic.

The biggest mistake in the "Build one to throw away" concept is that it implicitly assumes the classical sequential or waterfall model of software construction."

The problem is that you only will know what parts to throw away after the system is finished and the system testing is over. - ChristianOhman

1975: Basili, V. and Turner, A., Iterative Enhancement: A Practical Technique for Software Development :IEEE Transactions on SW Eng.

1981: Boehm, B., Software Engineering Economics Prentice-Hall. ISBN 0-13-822122-7 (pages 41-2, 254) allows for an iterative process when developing software.

1983: Booch, G., Software Engineering with Ada Benjamin-Cummings. (Around page 43) describes an iterative process for growing an object-oriented system.

1984: Madden, W and Rone, K., Design, Development, Integration: Space Shuttle Primary Flight Software System, CACM 27 9, Sept 1984, 914-925.

-- Although the publication was only in 1984, they used an iterative approach in 1977-79..

"An implementation approach was deivsed for STS-1, which met the objectives by applying the ideal cycle (they mean the waterfall cycle), to small elements of the overall software package on an iterative basis. ... STS-1 had 17 interim release drops in a 31-month period starting october 1977. Full software capability was provided after the 9th release in december 1978." - PhilippeKruchten

1984: Rzevski, G., Prototypes Versus Pilot Systems: Strategies For Evolutionary Information Systems Development, Approaches to Prototyping, Editors Budde et al, Springer-Verlag

1985: Boehm, B., A Spiral Model Of Software Development And Enhancement, 2nd. International Software Process Workshop. Coto de Caza, Trabuco Canyon, USA 1985. Wileden, J. and Dowson, M. (Eds.)

Notes: I'm not sure this citation is correct. - CraigLarman; PhilippeKruchten can offer this alternate:

1986: Barry Boehm, A Spiral Model of Software Development and Enhancement, ACM SIGSOFT Software Engineering Notes (SEN), August 1986

1985: Rzevski, G., Trends in Information Systems Design, Infotech State of the Art Review, Mature Systems Design, edited by L. Evans, Pergamon Press

1986: Currit, P. Allen, Dyer, Michael and Mills, Harlan D, Certifying the Reliability of Software IEEE TOSE, Vol. SE-12, No. 1, Jan86.

Notes: pp 3-11. Executable product increments are the basis for MTTF estimates. - TomGilb

1988: Gilb, T Principles of Software Engineering Management AW.

Notes: This had three chapters on Evolutionary Dev. - CraigLarman

1988: Edward, H BGen USA (ret.), Evolutionary Acquisition of Command and Control Systems: Becoming a Reality Signal, January 1988

Notes: pp 23-26 Found this reference in SPUCK93 (JPL, RDM) - TomGilb

1988: Boehm, B, A Spiral Model Of Software Development And Enhancement IEEE Computer. May 1988.

1991: Booch, G, Object-oriented Analysis and Design with Applications Addison-Wesley

Desribes a process for iteratively and incrementally growing a system.

1992: Ph. Kruchten, Un Processus de DúBloppement de Logiciel Itöâtif et Centrîdur l?Architecture 4×_ CongrÙrde Gõme Logiciel, Toulouse, France, Dï€mbre 1991, EC2, Paris

Iterative approach the "Rational way" (English version exists as a whitepaper from Rational) - PhilippeKruchten

1993: Cockburn, A, The Impact of Object-Orientation on Application Development IBM Systems Journal, 32(3), March 1993, pp. 420-444, reprinted in the 50-year anniversary issue, IBM Systems Journal 38(2-3), 1999. http://www.research.ibm.com/journal/sj38-23.html

Presents one view of the difference between incremental and iterative development (p. 311)

1996: Ph. Kruchten, A Rational Development Process Crosstalk, 9 (7) July 1996, pp.11-16.

(see [[1] (http://www.stsc.hill.af.mil/crosstalk/frames.asp?uri=1996/07/rational.asp)])

What will become the RUP lifecycle. - PhilippeKruchten

1996: Barry W. Boehm, 1996, Anchoring the Software Process IEEE Software, July 1996, pp.73-82.

where MBASE and RUP aligns concepts and terminology. - PhilippeKruchten

1996: Booch, G, Object Solutions Addison-Wesley.

Explains the importance and substance of an iterative and incremental lifecycle; talks about the growing of an architecture through successive refinement; introduces the notion of different rhythms (micro and macro process) in the lifecycle. About the same time there was a book by Adele Goldberg and Kenny Rubin of a similar nature. Alistair Cockburn's "SOOP" book followed shortly afterwards.

1998: Jennifer Stapleton, DSDM: Dynamic Systems Development Method Addison-Wesley

1998: Walker Royce, Software Project Management?A Unified Framework, Addison-Wesley-Longman

1999: Beedle, Mike; Devos, Martine; Sharon, Yonat; Schwaber, Ken; Sutherland, Jeff. SCRUM: An extension pattern language for hyperproductive software development. In Harrison, Neil; Foote, Brian; Ronhert, Hans (Eds.) Pattern Languages of Program Design 4. Addison-Wesley Software Patterns Series.

1992: Jacobson, Ivar, Object-Oriented Software Engineering: A Use Case Driven Approach. Chapter 2, The system life cycle.

In my edition, Addison-Wesley revised 1998, pp 23ff. --StevenNewton

See also Extreme Programming, kaizen