The first time I heard of “Agile development” I was already a pretty successful developer. My whole team was sent to several days of training. I liked what I heard and I thought I “got it.” What confounded me was that when my team tried to put this training into practice, no one seemed to have heard the same things that I had.

There is a general consensus that the optimal way to build software is using Agile processes — but that often seems to be where the agreement ends (if it even extends that far). A lot of teams salute at the Agile flagpole, but then march into battle using “waterfall” processes.

“Agile” is an umbrella term for several different practices — Scrum, Extreme, Kanban, etc. — derived from 2001’s The Agile Manifesto

As you can see from the six line manifesto, big project plans with robust documentation are the antithesis of Agile work patterns.

Less planning and murkier agreements are more effective at reaching our goals? Really?

Planning your Discoveries

Agile is a response to one critical insight: Building software is fundamentally exploratory. We often want to pretend that software can be planned out, that we can know in advance what problems we’ll face, and that we somehow know how long it will take to do something that has not been done before.

Development is unpredictable. You often hit bumps you didn’t expect, and sometimes you quickly resolve things you thought would be hard. In fact, if your software development does largely consist of rote processes, it’s long past time for you to update your tooling.

Confusion comes when we treat building software like traditional building — where we have clear blueprints in hand before we begin. Software development is unmapped territory: we can’t start at point A and simply march to the next milestone, we can only start at the beginning and try to work out a path to our goal — through occasional false starts and dead ends.

In a sense, Agile exists to enable businesses to manage this chaos. From the perspective of a traditional project manager, all this is evidence of poor planning. On your journey to Agile processes, however, don’t dismiss the project manager’s concerns prematurely. They still have a role and responsibility to the customer.

Agile teams are organized to deal with this essential unpredictability. Both a jazz combo and a classical orchestra can be effective organizations in their way, but change and unpredictability will engage one and derail the other. 

eautiful things can happen when a highly skilled team carefully follows a score. But it is an expensive and inefficient way to do software development.

Planning a Project

Let’s take a simple, straightforward problem: try to come up with a reliable estimate on how long it would take to clean out this garage.

The first thing you’ll notice is that you can’t even see all the issues you’ll confront. You’ll have to go digging around and moving stuff just to be able to start planning. You’ll have to start making an inventory of the types of items that you have to move then apply an estimate for each type. Ultimately, you may have to do a mental walk-though of the entire effort just to create the estimate.

If we worked at it long enough, we would be able to come up with a checklist of all tasks that needed to be complete in order to completely clean the garage. This approach essentially places all of the “discovery” effort into the planning stage.

The process of creating and executing a big project plan is called “Waterfall”, after the cascading bars on a Gant chart.

Lean Principles

I need to interrupt myself and talk about “waste”.

Recent Agile methods have returned more aggressively to Agile’s roots in Lean manufacturing. At bottom, Agile is about using Lean principals to organize around managing change. And Lean is all about managing “waste”.

Agile practices give us the “how”, while Lean principles explain the “why”.  

There are far too many teams that have adopted Agile and seen little benefit. This is because they adopted Agile processes without really eliminating waste.

In the garage example, you may have noticed that all the estimation and scheduling work did not get us any closer to a clean garage! From a Lean/Agile perspective, all of the effort involved in creating a big project schedule is “waste” because it is added no value to the finished work.

A customer once called me to a meeting to discuss the trouble he was having generating the delivery forecasts needed for his team. I happened to bring a vendor along who asked, “How accurate are these reports?”. The customer replied that they were right more than half the time. The vendor then asked, “Since you admit that can’t trust these reports, why are you doing them at all?”

There is a good chance that people already consider your project plans a waste. The irony is that the more energy you put into trying to fix their unreliability, the more formal Lean “waste” you create (work that does not contribute to the final product is waste).

Remember that the value that development teams deliver is new business capabilities. Great planning, progress reporting, architecture, etc. only have value as they support that deliverable.

Big plans also mean big batches of requirements, which tends to hide priorities. Delivering features that are not priorities to the business is another form of Lean “waste”.

In short, when creating a well-documented delivery schedule has itself becomes a business accountability, our incentives have become misaligned. We shouldn’t have incentives to create waste, especially when Agile provides a Leaner, more effective, and more organic approach.

The Agile Approach

Using an Agile process, we would block out, say, a 4' by 4' section and just start organizing there. When that section was done, we would pick another section and start again. Any unexpected difficulties we uncover would be dealt with as they were uncovered. 

In other words, the Agile approach is to: do something, adjust, repeat. Designs are high-level, if they exist at all; project schedules are only estimates.

With an Agile approach, if the cleaning isn’t being done the way the owner expected, this can be caught and corrected as soon as the first small section is cleaned — rather than after the entire master plan is executed. Even without a “master plan”, the owner can see regular progress, so the owner has visibility to when the whole effort will likely be complete. And if we were to discover something incredibly valuable — or dangerous — hidden in this mess, the team can easily adjust their approach without derailing a project plan.

Best of all — and this is important — the owner will start to enjoy the benefit of a partially cleaned garage almost immediately!

Process Comparison
Planning Centric Agile
• Owner measures progress according to the plan’s milestones • Owner sees progress via incremental delivery
• Owner expectations must be fully captured in the plan (which encourages pile-on requirements) • Owner expectations are re-validated with each quick release
• Changes require re-planning • Changes are non-disruptive
• Owner can use fully cleaned garage when project is complete • Owner enjoys the benefit of a partially clean garage even while project is underway

An Agile approach requires a rethink about how a team manages delivery, change, and complexity.

Managing Delivery

Transitioning away from Waterfall-style planning can still meet the business’ desire to understand and manage the progress of the deliverable. But it does so organically. A great visualization for customers to understand a team’s progress toward feature delivery is with a feature “burn down” chart.

A burndown chart shows team progress in clearing the “user stories” (small units of functionality) in the Feature backlog. Since the Feature consists of these constituent parts, it illuminates a likely Feature completion date without the need for complex project management. As more work is done, the projected completion date becomes more accurate. This allows business partners to make plans, without engaging the development team in wasteful project tracking. 

Managing Change

Our incomplete knowledge is always a risk to the Big Project Plan, but the project plan itself may be an even bigger risk.

Any type of change can derail carefully planned projects, including:

  • Changes due to mis-communicated requirements
  • Evolving understanding of the usefulness of different features
  • Changes in business circumstances or strategies

If a customer request generates a formal change-order process (re-estimation, requirements updates, new documentation, etc.) the team’s incentives will be to resist change. However, notice that each of the changes listed above are valid, maybe critical. The team needs to be structured so that it’s easy to adapt to business agility.

Because change happens over time, the best protection against this risk is fast delivery! Delivering small, fast units of functionality without project overhead allows everyone to adapt. As it happens, this is another core Lean/Agile principle.

Managing Complexity

Even if there were no risk of dealing with change, working iteratively on small problems on a fast delivery cycle is also generally the best way to manage complexity.

Lean explicitly encourages teams to delay decisions until the most information is known. Upfront planning, of course, forces decisions when the least information is known.

This approach:

  • Allows you to learn as you build
  • Helps avoid over-building, and
  • Enables more systematic testing.

Of course, there are also considerable overhead and impedance concerns with communicating big plans to teams. Smaller, faster iterative deliveries have been shown to be generally better at even complex solutions. 

Next

Within the software development community, there is generally a widespread appreciation that Big Plan “waterfall” methodologies are not an optimal way of building software. In fact, the stigma of Waterfall is such that most teams would see the label as pejorative. In terms of actual practice, however, in my experience only a minority of teams manage to really do Agile development.

If Agile is such a transformative way of understanding and managing the software development process, why it is so often implemented badly? I’ll examine why so many teams are unable to transition to Agile processes in my next post.