The Curse of Traditional Software Development Methodologies

I've realized that very few teams follow the waterfall methodology by the book. However, I think that most shops try...really, really hard. Of course we WANT to define every requirement and scenario that could possibly happen. Once that's complete, THEN we can start building. And, if there is a new scenario, or "missed" requirement (heaven forbid), then the developers can point at the requirement gatherers and say "AH - HA!!! You screwed up, and now you must suffer the consequences!".

Well, this is just silly, we all know that change happens, but yet we try to fight it every step of the way. That's what requirements change management is, right? Its to prevent change, i.e. just say NO! To remain competitive in today's marketplace, we just can't think like this anymore.

Below is a list of beliefs that we (including myself) really wish was true. If they were true, our jobs would be so easy! Of course, it wouldn't be as fun :-)

• The customer knows exactly what they want.
  
• We understand what the customer wants.
  
• We understand who the customer is.
  
• We can define everything up front.
  
• The technical team knows how to build what the customer wants.
  
• We can accurately predict how long it will take to develop what the customer wants.
  
• There won't be any problems (or very few).
  
• Documentation is a true measure of progress.
  
• Nothing should change. If it does, it's [fill in the blank] fault.
  
• Hours worked is infinitely directly proportional to value delivered.
  

Okay, so none of these things are really true. Since we all like to pretend, we have failed over, and over, and over again. I believe that this failure in traditional projects has caused stereotypes to evolve, including:

• The technical team is arrogant and lazy.
  
• The technical team doesn't care about the customer.
  
• The technical team is lying. It can't be as hard as they say.
  
• The customer doesn't respect the amount of work necessary to deliver what they want.
  
• The customer is stupid.
  

This failure has also caused some pretty weird behaviors to spring up.

• A tendency to over-specialize and divide ourselves into silos.
  
• The desire to control and direct every move made by every team member.
• Forgetting that development is creative and innovative and can't be treated as a commodity (this causes some real bad implementation).
• A culture of command and control.
• The desire to create a process for EVERYTHING in an attempt to control.
• A fear of sharing "too much" information with other silos...I mean other teams. If we share "too much" information, then the other teams may want "change".
• A fear of sharing too much information with the customer. Again, for a fear of change.

Please, stop the madness. Let's stop pretending that traditional development works and start working together.

Hard Questions for Agile

I am seeing a lot of the same questions come up over and over again recently. Now that larger organizations are starting to see the benefits of agile, more and more questions are coming up.

Everyone (including myself) has a desire to have a prescribed method, process, checklist, etc. that they can use to make sure they are doing things the "right way". In Scrum, there is a pretty clear list of activities, artifacts, and roles. People struggle because these lists provide the "whats" but not the "hows". For example, when I tell teams that they need to have one product owner for that project, they are sometimes at a loss; "How do we find one product owner? Our organization won't support that!".

Below is a list of questions that I have seen come up over and over again. I'm going to address them over time (as I have more time).

1. How can agile work on large projects?
• I know this is being addressed slowly, but there are just not enough case studies out there. The standard "Scrum of Scrums" just doesn't cut it anymore, meaning that there are serious complexities that aren't addressed in the standard definition.
2. How do I work with a vendor that does not want to take and agile approach?
3. How does interaction design work in an agile environment?
   
• I have not met any interaction designers (HCI type people) who really buy into the concept of iterative, incremental development.
4. Our company is committed to PMBOK, how can agile fit in this environment?
5. Our company needs to make financial projections that are 5 years out. How can we do this using an agile approach without trying to plan everything up front?
   
• We in agile know that it is a myth that you can accurately plan everything up front.
6. Our company needs to be CMMI compliant to be able to work with government entities. How can we be CMMI level 1-5 compliant and still follow the agile values?
7. We have a project that's in crisis. How can agile rescue our project?
   

I'll add more as I hear them, but these are the biggies. Feel free to provide input or links that addresses any of these questions.

Agile won't work on [fill in the blank]

I have four kids ranging from 15 to 3. For some reason, people with no kids really like to give my wife and I advice on how to raise kids. Usually they will provide analogies using their favorite pet. Or, they will tell me about their second cousin twice removed who "just had a baby", and here's what they PLAN on doing. I just politely nod and say "thank you".

So, what's the point? Well, I tend to get a lot of advice, or "words of wisdom" regarding agile development and how it won't work on certain projects. I've heard it won't work on complex projects, big projects, geographically disperse projects, mission critical projects, non-software projects...you name it. Everyone who tells me this has either a) never worked on an agile project or b) worked with a poor implementation of agile.

Instead of politely nodding and saying "thank you for your advice" as when I'm given child-rearing advice, I've recently decided to challenge these ideas. Of course not in a confrontational way, but just in an inquisitive way. "What experiences have led you to that belief?". "Why can't an offshore project implement agile methodologies?".

Below is a post that I saw on LinkedIn:

"As with any project management tool, it is situational. If, like in many software development environments, there are one or two key people involved in each and every task, Agile tends to break down. In this case, there are better methodologies to employ. Like any concept, I think the rub is that no technique fits all situations."

This is clearly someone providing advice on agile development that clearly has never worked in an agile environment, or has suffered a poor implementation. I couldn't resist responding. Below is my post.

"There is a very broad spectrum of agile methodologies out there that will cover any situation. Being agile also means adapting the process to fit your needs as it's an empirical, "inspect & adapt" approach to managing projects. For instance, typically Scrum & XP are implemented together. Scrum only addresses project management issues, while XP addresses development issues.

The failed agile implementations I have seen were caused by one or more of the following:

1. The team is not empowered and self-organizing.
2. Team members are working on too many projects, resulting in excessive context switching.
3. Management is committed to command and control (related to #1)
4. The customer or customer representative is not involved.
5. Processes are not allowed to evolve.
6. There is no dedicated product owner, i.e. someone responsible for the ROI that the team can go to (getting rid of the multiple boss syndrome).
7. The ones leading the agile implementation have not been properly trained in agile.

So, agile breaks down when the organization is not ready to accept its principles. If the organization is not willing to change from command & control to servant leadership, empower the team, assign ONE product owner, and get the customer involved, then it will fail."

Hopefully I didn't come off as a jerk on this. I just couldn't resist.

If you feel that agile just won't work on _____, I encourage you to find out for yourself. There are plenty of cases where it has worked on huge, complex, mission critical projects.

Sprint Planning When Everyone's New

We were working on a legacy product written in PowerBuilder. There was one engineer that had been around for a while, about 6 years. Another had been around for a little less than a year, and the rest were brand new. The entire QA team was new. They weren't only new to that project, they were new to QA. There was also a team of business analysts. Most of them had been around for several years, but there were only 2, and this was a big project.

As to be expected on teams such as this, the process was very waterfall-ish and chaotic. The team waited for the BAs to create "the specs", and QA waited for engineering to do the coding. Lots of waiting.

It was my job to implement Scrum at this company. We finally had a product backlog that was prioritized, but that was about it. Even though that was such a small accomplishment, it made a very positive impact on the team. Before this, the team never new what to work on next, and usually worked on things that had to be abandoned later.

So, we have a product backlog. We still hadn't conducted a release planning or a sprint planning session. I decided to start with a sprint planning session.

Sprint Planning Session 0 = Miserable Failure

Our first sprint planning session was a disaster. First of all, the QA team did not show up. I almost just canceled right there, but I thought I'd continue this disaster.

We projected the product backlog (which was in excel) and started going down this list. The goal was to define "Done" for each item, and also to decompose it into smaller tasks, with each task less than 16 hours.

We got stuck on the first item. Everyone said "I'll have to go back and look at the spec" for every item we brought up. And if there wasn't a spec, it was just impossible. Everyone just stared at each other like deer in the headlights.

I also heard the engineers say things like "we'll have to go into the code, it will take a couple of days to break this down". Or "there is no way we can break down this 180 hour task" (yes, they had estimated the product backlog items in hours).

We canceled the meeting about 15 minutes into it. While I was very disappointed that it failed, I was excited to have a new challenge. In addition, it was great that I learned something about the inner-workings of the team. How do you conduct a sprint planning session when no one knows how to implement the features in the backlog?

Sprint Planning Session 1 = Moderate Success!

I was sure that the team just needed to do homework. But, after talking with the old timer, I realized that homework was not enough!

The code base was extremely unstructured. It had grown over the last 12 years into something really ugly and barely maintainable. Not even the engineer with the most experience would have been able to come up with sprint tasks that were 16 hours or less. I got some great advice from someone on the yahoo scrum group, about maybe using spiking to help decompose the backlog items. While I love the idea, there was no way we had time to do this on every item.

So, here's what we did. We encouraged team to do some research on the upcoming features.

Then, in the sprint planning session, the team self-organized into smaller teams, and dug in deep on the features. We printed specs out for them, and of course had the BAs there (which were ultimately playing the role of product owner).

The engineers had their laptops so they could look into the code if necessary, I suppose doing "mini-spikes".

This time QA showed up, but they were there in bodies and not in spirits. But, that's a subject for another post.

We still did NOT come up with a sprint backlog. I was a little disappointed in that. However, the team came out feeling like a true team, and with a clearer understanding of the sprint ahead of them. We also had a clear commitment (which the business broke, again, for another post).

It was a definite step in the right direction. While we weren't able to derive the sprint backlog, the team bonded during the exercise. The team also understood what they needed to deliver (they just weren't sure how at that point). In addition, we realized where our problems truly existed.