When learning about Scrum and agile, iterative & incremental development is among the first expressions we come across. Actually, these are two different concepts, let’s see what they mean.
Iterative Development
As we know, iteration is a word of Latin origin, meaning doing something repeatedly. In mathematics, it denotes an approach with which we can proximate a value. We can apply this concept to the design process, too, because our work rarely reaches the intended goal perfectly on the first attempt. The more complex the task is, the more likely we have to correct our initial plans. That is to say, the consecutive changes lead us closer and closer to the desired outcome. Moreover, even if we think our plan is perfect, it needs validation. The ultimate validation is the implementation process and the acceptance testing of the implementation. An iterative approach acknowledges that our initial plans are likely to be imperfect. Therefore, it is reasonable to plan the revision of our design as we learn more during the implementation process. Iterations do not necessarily result in increments, in the sense that the solution does not have to be in a usable condition at the end of an iteration. The iteration does not necessarily serve a business purpose, it is rather an engineering technique.
Applying an iterative approach is not a call for an ignorant and poor initial design. When we purposefully apply iteration, it is still desirable to take into consideration what is already known. It is not an excuse for working without any plan.
Incremental Development
As the name indicates, incremental development delivers a solution in increments, building blocks, and pieces. Every increment adds new features, capabilities, qualities, and components to the solution. Not every increment is made with the intention of delivering a product ready for use. There is no strict definition of an increment, it can be a separately testable feature just as well as a quarterly release. The main point is the birth of a new ‘unit’ which adds value to a solution. If the increments are (more or less) independent from each other, it is possible to develop them in parallel, by different actors.
Iterative & Incremental Development
Combining the iterative and incremental approach is not only possible but results in a new quality. Our instinct would tell us that doing things right at the first attempt is the most economical, fastest and best approach. Hence, iterating means waste, the less iteration is needed the better off we are. Therefore it is difficult to accept that iterating is desirable, as opposed to blaming someone for the need to correct already delivered solutions. However, delivering a solution in pieces not only justifies iteration but opens new opportunities since the iteration serves the purpose of an increment. For example, the concept of Emergent Architecture suggests designing a solution ‘just enough, just in time’. Just enough means that the solution does not have to consider potential future requirements. Just in time means that it is done in the ‘last responsible moment’. This approach has a multitude of benefits, for instance, the design will certainly serve real and known requirements, will rely on what has been implemented, will be designed by people who actually implement it, can apply the latest and best available technologies, etc.
Agile Iterative & Incremental Development
Iterative and incremental development is not necessarily agile. The point of agile is achieving agility: business should be in a position to choose from multiple options at any time: continue the development of a product, call off the rest of the project yet realize benefit, define a new course for the development, reprioritize deliverables, etc. In other words, should be able to take rapid actions. Iterative & incremental development can help achieve this goal, provided that every cycle ends in a clear situation. That is to say, the end result must be a useable, potentially releasable product, not needing further (hidden) work. This explains why is it important that user stories should fit into a Scrum Sprint, and why test-driven development is a popular technique for agile teams. In contrast, if the technical debt is growing from iteration to iteration, the organization will not be in a position to make the aforementioned decisions.
The agile way has apparent benefits. Organizations can release solutions early and hence start realizing benefits early. Having a working solution early eliminates the risk of ending up with nothing but costs. Users can give feedback from the early stages and throughout product development. Stakeholders can rely on what is known and ask for changes. That is to say, even ‘try and err’ comes with benefits. On a side note: in Agile we rather work on products and not projects. A product hits the market early and receives constant improvements throughout its lifetime to avoid obsoletion.
The Traditional Approach
Traditional project management reached its maturity many decades ago. It proved to be quite successful in large construction projects. This approach enables planning all activities and resource uses of a complex, large endeavour with an optimized schedule and with risk management techniques. Project management is industry-agnostic and applicable everywhere, including in software development. However, the specific needs of software development make the traditional approach suboptimal.
The traditional project management approach organizes the work as a sequence of one-off activities:
Based on its visual representation it is also called the Waterfall approach. In the Waterfall approach, every activity is executed once, and the consecutive activities are planned with the assumption that the dependencies will be delivered in time (more or less), in a flawless condition. Any findings having an impact on the already delivered pieces of work mean a serious problem. That is to say, the delivery process needs to jump back to a previous stage for a fix and flow through the waterfall again before it can continue. As a consequence, any change request has a detrimental impact on the project, with probable rescheduling and re-budgeting.
Delivery is a one-off, nothing is available for consumption or evaluation before the very end of the project. All resources are locked in for a long time and there are no viable options to alter the course of delivery whatever happens in the environment. The project delivers ‘everything or nothing’. For all these reasons the failure rate of waterfall projects is relatively high.
However, a pure waterfall approach is rarely applied. There are different techniques for reducing the risks of a traditionally managed project, e.g. delivering quarterly releases would give a certain degree of agility. Slicing the project to business phases and milestones, each delivering a tangible, usable result would definitely reduce the above-mentioned risks. Development normally follows a priority list, leaving room for scope versus deadline decisions.
We have to recall that in the early years of software development, the technical conditions of Agile were in the realms of dreams. Running a code had a much bigger cost, therefore programmers worked ‘offline’, trying to sort out all potential issues very rigorously before the first attempt of compiling their code, not to mention executing it. It took time to come up with all the protocols, languages, frameworks, standard tools and techniques we use today. The effort needed to implement complex functionality is incomparably lower today than 4-5 decades ago.
There are different techniques for reducing the risks of a traditionally managed project, e.g. delivering quarterly releases would give a certain degree of agility. Slicing the project to business phases and milestones, each delivering a tangible, usable result would definitely reduce the above-mentioned risks. Development normally follows a priority list, leaving room for scope versus deadline decisions. This evolution of project management resulted in the change of paradigm that led to the emergence of Agile with new concepts and methods better addressing the shortcomings of the pure waterfall.
The Self-managing – Cross-functional Agile Teams
In a traditional organizational model people sit together with experts in the same area, e.g. database developers and business analysts in their own respective groups. They receive their tasks from their managers, or in a more modern matrix environment from a project manager. Developers receive detailed instructions from an architect, testers receive detailed test cases, etc. This model makes sense in a bureaucratic organization, however, it has many drawbacks in a product development environment. We should expect to see:
- Comparatively poor communication, slow turnover, misunderstandings, strained relationships.
- Lost talent since everything is organized top-down, with no opportunity to channel the knowledge of the broader team.
- Low motivation levels, since people only follow instructions with little autonomy, creativity, and responsibility.
The Agile model solves the above problems, and also enables the teams to create ‘vertical slices’. A vertical slice is a complete, functional piece of a system, whatever is needed to be added in e.g., the front-end, the back-end, the data, etc. As we can see, a vertical slice has a lot in common with an increment.
Recommended exercise: Watch the Factory Scene from Charlie Chaplin’s Modern Times