The Curse of Traditional Software Development Methodologies

I've realized that very few teams follow the waterfall methodology by the book. However, I think that most shops try...really, really hard. Of course we WANT to define every requirement and scenario that could possibly happen. Once that's complete, THEN we can start building. And, if there is a new scenario, or "missed" requirement (heaven forbid), then the developers can point at the requirement gatherers and say "AH - HA!!! You screwed up, and now you must suffer the consequences!".

Well, this is just silly, we all know that change happens, but yet we try to fight it every step of the way. That's what requirements change management is, right? Its to prevent change, i.e. just say NO! To remain competitive in today's marketplace, we just can't think like this anymore.

Below is a list of beliefs that we (including myself) really wish was true. If they were true, our jobs would be so easy! Of course, it wouldn't be as fun :-)

• The customer knows exactly what they want.
  
• We understand what the customer wants.
  
• We understand who the customer is.
  
• We can define everything up front.
  
• The technical team knows how to build what the customer wants.
  
• We can accurately predict how long it will take to develop what the customer wants.
  
• There won't be any problems (or very few).
  
• Documentation is a true measure of progress.
  
• Nothing should change. If it does, it's [fill in the blank] fault.
  
• Hours worked is infinitely directly proportional to value delivered.
  

Okay, so none of these things are really true. Since we all like to pretend, we have failed over, and over, and over again. I believe that this failure in traditional projects has caused stereotypes to evolve, including:

• The technical team is arrogant and lazy.
  
• The technical team doesn't care about the customer.
  
• The technical team is lying. It can't be as hard as they say.
  
• The customer doesn't respect the amount of work necessary to deliver what they want.
  
• The customer is stupid.
  

This failure has also caused some pretty weird behaviors to spring up.

• A tendency to over-specialize and divide ourselves into silos.
  
• The desire to control and direct every move made by every team member.
• Forgetting that development is creative and innovative and can't be treated as a commodity (this causes some real bad implementation).
• A culture of command and control.
• The desire to create a process for EVERYTHING in an attempt to control.
• A fear of sharing "too much" information with other silos...I mean other teams. If we share "too much" information, then the other teams may want "change".
• A fear of sharing too much information with the customer. Again, for a fear of change.

Please, stop the madness. Let's stop pretending that traditional development works and start working together.

Hard Questions for Agile

I am seeing a lot of the same questions come up over and over again recently. Now that larger organizations are starting to see the benefits of agile, more and more questions are coming up.

Everyone (including myself) has a desire to have a prescribed method, process, checklist, etc. that they can use to make sure they are doing things the "right way". In Scrum, there is a pretty clear list of activities, artifacts, and roles. People struggle because these lists provide the "whats" but not the "hows". For example, when I tell teams that they need to have one product owner for that project, they are sometimes at a loss; "How do we find one product owner? Our organization won't support that!".

Below is a list of questions that I have seen come up over and over again. I'm going to address them over time (as I have more time).

1. How can agile work on large projects?
• I know this is being addressed slowly, but there are just not enough case studies out there. The standard "Scrum of Scrums" just doesn't cut it anymore, meaning that there are serious complexities that aren't addressed in the standard definition.
2. How do I work with a vendor that does not want to take and agile approach?
3. How does interaction design work in an agile environment?
   
• I have not met any interaction designers (HCI type people) who really buy into the concept of iterative, incremental development.
4. Our company is committed to PMBOK, how can agile fit in this environment?
5. Our company needs to make financial projections that are 5 years out. How can we do this using an agile approach without trying to plan everything up front?
   
• We in agile know that it is a myth that you can accurately plan everything up front.
6. Our company needs to be CMMI compliant to be able to work with government entities. How can we be CMMI level 1-5 compliant and still follow the agile values?
7. We have a project that's in crisis. How can agile rescue our project?
   

I'll add more as I hear them, but these are the biggies. Feel free to provide input or links that addresses any of these questions.

Agile won't work on [fill in the blank]

I have four kids ranging from 15 to 3. For some reason, people with no kids really like to give my wife and I advice on how to raise kids. Usually they will provide analogies using their favorite pet. Or, they will tell me about their second cousin twice removed who "just had a baby", and here's what they PLAN on doing. I just politely nod and say "thank you".

So, what's the point? Well, I tend to get a lot of advice, or "words of wisdom" regarding agile development and how it won't work on certain projects. I've heard it won't work on complex projects, big projects, geographically disperse projects, mission critical projects, non-software projects...you name it. Everyone who tells me this has either a) never worked on an agile project or b) worked with a poor implementation of agile.

Instead of politely nodding and saying "thank you for your advice" as when I'm given child-rearing advice, I've recently decided to challenge these ideas. Of course not in a confrontational way, but just in an inquisitive way. "What experiences have led you to that belief?". "Why can't an offshore project implement agile methodologies?".

Below is a post that I saw on LinkedIn:

"As with any project management tool, it is situational. If, like in many software development environments, there are one or two key people involved in each and every task, Agile tends to break down. In this case, there are better methodologies to employ. Like any concept, I think the rub is that no technique fits all situations."

This is clearly someone providing advice on agile development that clearly has never worked in an agile environment, or has suffered a poor implementation. I couldn't resist responding. Below is my post.

"There is a very broad spectrum of agile methodologies out there that will cover any situation. Being agile also means adapting the process to fit your needs as it's an empirical, "inspect & adapt" approach to managing projects. For instance, typically Scrum & XP are implemented together. Scrum only addresses project management issues, while XP addresses development issues.

The failed agile implementations I have seen were caused by one or more of the following:

1. The team is not empowered and self-organizing.
2. Team members are working on too many projects, resulting in excessive context switching.
3. Management is committed to command and control (related to #1)
4. The customer or customer representative is not involved.
5. Processes are not allowed to evolve.
6. There is no dedicated product owner, i.e. someone responsible for the ROI that the team can go to (getting rid of the multiple boss syndrome).
7. The ones leading the agile implementation have not been properly trained in agile.

So, agile breaks down when the organization is not ready to accept its principles. If the organization is not willing to change from command & control to servant leadership, empower the team, assign ONE product owner, and get the customer involved, then it will fail."

Hopefully I didn't come off as a jerk on this. I just couldn't resist.

If you feel that agile just won't work on _____, I encourage you to find out for yourself. There are plenty of cases where it has worked on huge, complex, mission critical projects.

Sprint Planning When Everyone's New

We were working on a legacy product written in PowerBuilder. There was one engineer that had been around for a while, about 6 years. Another had been around for a little less than a year, and the rest were brand new. The entire QA team was new. They weren't only new to that project, they were new to QA. There was also a team of business analysts. Most of them had been around for several years, but there were only 2, and this was a big project.

As to be expected on teams such as this, the process was very waterfall-ish and chaotic. The team waited for the BAs to create "the specs", and QA waited for engineering to do the coding. Lots of waiting.

It was my job to implement Scrum at this company. We finally had a product backlog that was prioritized, but that was about it. Even though that was such a small accomplishment, it made a very positive impact on the team. Before this, the team never new what to work on next, and usually worked on things that had to be abandoned later.

So, we have a product backlog. We still hadn't conducted a release planning or a sprint planning session. I decided to start with a sprint planning session.

Sprint Planning Session 0 = Miserable Failure

Our first sprint planning session was a disaster. First of all, the QA team did not show up. I almost just canceled right there, but I thought I'd continue this disaster.

We projected the product backlog (which was in excel) and started going down this list. The goal was to define "Done" for each item, and also to decompose it into smaller tasks, with each task less than 16 hours.

We got stuck on the first item. Everyone said "I'll have to go back and look at the spec" for every item we brought up. And if there wasn't a spec, it was just impossible. Everyone just stared at each other like deer in the headlights.

I also heard the engineers say things like "we'll have to go into the code, it will take a couple of days to break this down". Or "there is no way we can break down this 180 hour task" (yes, they had estimated the product backlog items in hours).

We canceled the meeting about 15 minutes into it. While I was very disappointed that it failed, I was excited to have a new challenge. In addition, it was great that I learned something about the inner-workings of the team. How do you conduct a sprint planning session when no one knows how to implement the features in the backlog?

Sprint Planning Session 1 = Moderate Success!

I was sure that the team just needed to do homework. But, after talking with the old timer, I realized that homework was not enough!

The code base was extremely unstructured. It had grown over the last 12 years into something really ugly and barely maintainable. Not even the engineer with the most experience would have been able to come up with sprint tasks that were 16 hours or less. I got some great advice from someone on the yahoo scrum group, about maybe using spiking to help decompose the backlog items. While I love the idea, there was no way we had time to do this on every item.

So, here's what we did. We encouraged team to do some research on the upcoming features.

Then, in the sprint planning session, the team self-organized into smaller teams, and dug in deep on the features. We printed specs out for them, and of course had the BAs there (which were ultimately playing the role of product owner).

The engineers had their laptops so they could look into the code if necessary, I suppose doing "mini-spikes".

This time QA showed up, but they were there in bodies and not in spirits. But, that's a subject for another post.

We still did NOT come up with a sprint backlog. I was a little disappointed in that. However, the team came out feeling like a true team, and with a clearer understanding of the sprint ahead of them. We also had a clear commitment (which the business broke, again, for another post).

It was a definite step in the right direction. While we weren't able to derive the sprint backlog, the team bonded during the exercise. The team also understood what they needed to deliver (they just weren't sure how at that point). In addition, we realized where our problems truly existed.

Struggles of the New "Agile" Consultant

In the Beginning, There Was Chaos

I've always wanted to be someone who mentored and taught others how to live up to their potential. Naturally, I wanted to become a consultant. The path that lead me to where I am will have to be the subject of another post. For now, I'll just summarize.

After college, I went in to retail. What else could you do with a Psychology degree? After about four years in retail management, I went back to college to get a Computer Science degree. I had various jobs where I performed the functions of systems engineer, software engineer, DBA, SEO, etc. Then, I graduated to development management where I managed a couple of teams at a large software company. Surprisingly, managing a software development team isn't that much different that managing a retail team.

While managing these teams, it was hard to ignore the fact that they were in total chaos. My company was pushing a home-grown agile software development methodology. It sounded good because there was more focus on the customer, but it just didn't work. So, I went on the journey of finding what agile development meant. Certainly it didn't mean "no documentation" and "no design", right? Well, it didn't, but that's how we were acting. The teams were working super long hours. The engineers had no idea how to estimate. The team really didn't know what was supposed to be delivered.

Let There Be Scrum!

Then, slowly through several "iterations", we implemented scrum. I played the Scrum Master role. The path we took could be mapped by books that were used. The books that paved the road for us were "Agile Estimating and Planning", by Mike Cohn, then "User Stories Applied", by Mike Cohn, and then finally "Agile Software Development with Scrum", by Ken Schwaber. As we went through each book, we implemented these new things we learned incrementally.

So, we did it. We successfully implemented scrum. It was awesome. Why was it awesome? Here are some of the key reasons why:

• Freedom - The team was free to do whatever it took to make software. The management team didn't care if we used Scrum, XP, or Waterfall, they just wanted software. We were continually getting beat up for the crap we turned out. We finally got tired of it, and management let us do whatever it took.
• Engaged Teams - I was blessed with a wonderful team. Was every single person on the team engaged and wonderful? NO. But, these things have a way of working themselves out. The team, with a little guidance, quickly became a high performing, self-organizing team.
• Awesome Product Owner - Our product owner on our team was amazing. She was strong, had technical sense, and strived to play by the Scrum rules. She mastered the product backlog. Sure, maybe at times she was a bit "too" engaged throughout the sprint, which caused minor distractions at times, but that was to be expected and easily corrected.
• Individuals and Interactions Over Processes and Tools - You probably recognize that this is part of the agile manifesto. I listed this explicitly because this was a big deal. Originally, we were trying to use all kinds of tools; QuickBase, TeamTrack, TestDirector, Excel, etc. It was becoming a mess. We were spending so much time making sure everything was up to date. In addition, we were spending time generating reports that no one was reading. In the end, we used the tool with the lowest possible impact to the team; the white board. The teams sprint backlog was documented on the white board, where we tracked the daily status.
  
  Okay, to be honest, we didn't "just" use the white board. We also used a tool called "ExtremePlanner" to hold the entire product backlog, and help with sprint and release planning. This was basically the playground for the product owner, so there was no overhead experienced by the team.

Don't get me wrong, the road wasn't always smooth. There was a lot of history that we had to work through. There was a pile of crap that we called software that was released to the public, and we had to fix it. But, we did.

Enter the Agile Consultant

I really wanted to use my experiences over the last couple of years, both good and bad, to help others make the transition to common sense, i.e. Scrum. I know that most software development shops really didn't know how to make software that the customer wants without smashing the deadline, scope, or quality. That's exactly where we were.

Our company closed our office in Omaha, and I didn't want to move to the home base in San Diego, CA. This was the perfect opportunity to step into an "agile" consultant role. By certain divine intervention, a local company was looking for a consultant who was an expert in agile software methodologies. That was my calling.

My first job as a consultant is working with a company that was (and still is) in trouble. Some promises were made by executives that are now long gone, and the current team has to make good on those promises. Below is a sample of some of the core problems at this division:

1. Legacy software - Most companies who have been around for 5+ years have to deal with legacy software that is fragile and extremely time consuming to make any changes. This is just about the worst case that I've seen or even read about. The code is so poorly architected it is a significant challenge to make even the smallest of changes.
2. Turnover - Approximately 95% of the original team left. Most of the people who left held knowledge that was not documented nor was it shared. Just about everyone working on this project is new (within 6 months).
3. Poor development infrastructure - The team is using VSS, which performs file-locks when code is checked out. This is a serious bottle-neck. Due to the poor design of the code, there are only a few files that most programmers have to touch. In addition, there is no consistent build or test environments or process.
4. Lack of core experience - There are some key personnel, such as BA, QA, Management, that have no experience in those disciplines. Not only do they have to learn the product, they have to learn how to do their job.
5. Micro-Management - There is a desire to manage every last task the team has to do. This has had a very damaging effect on the team. Most of the team is paralyzed when they have to make any decision for themselves. In addition, the management style is militant and somewhat cold.
   

I knew that this would be a challenge going in. We changed our approach several times. Below are some things we tried.

1. Implement Scrum on a small project and use the success to convince others to adopt. We would play roles on the Scrum team, product owner and Scrum Master.
   
2. Educate everyone on Scrum, then let them try it themselves on any product they decided, with us helping and coaching only. We wouldn't play any roles on the scrum teams.
   
3. Implement Scrum on the flag-ship product, and then let the other smaller projects fall in line. We would stay very close and coach the new Scrum Master, product owner, and team.
   

We actually did all of these things in order, and are now on #3. Scrum is well underway.

We went through quite a few struggles that we weren't expecting.

• We assumed that everyone would welcome us with open arms and that there would be a high level of trust. Boy, was I wrong. Most people were quite resistant. Many have come around after seeing some of the small successes, but there are some that I see that will always be resistant.
• We underestimated the learning curve of Scrum. Yes, Scrum is simple. But, as Ken Schwaber (one of the original creators of Scrum) frequently states, "Its incredibly difficult to implement". For example, creating a list of prioritized work has proven to be incredibly difficult. The product owner believed (for a while) that everything had the same priority ... HIGH! We are planning on conducting additional training sessions to accommodate.
  
• We didn't know how much trouble the project was really in. As we all know, not all projects can be saved. It doesn't matter what process you use. Again, as I've heard Ken Schwaber say, with Scrum, you'll know how screwed you are after 30 days. This couldn't be more true.
• There is no trust amongst the team. QA doesn't trust development, development doesn't trust QA, BA doesn't trust anyone, etc. What has been incredibly helpful here is our daily scrums. Just some time for the entire team to get together every day has helped the team members come closer to a trust relationship.
• People don't understand what self-organizing and cross-functional mean. The concept that you would have people from different disciplines on the same team is crazy! What's more crazy is that you would allow the team to work on what they want to. When we introduced this concept, it was like we just told them they should believe in elves. I was shocked. To me, these concepts are common sense for any good manager. As Scrum is working, and we are coaching everyone along the way, these concepts aren't approached with dismay and confusion anymore. It will be a slow process, but I think that the team will come around.
  

I've learned a lot while implementing Scrum at my first gig as an agile consultant. It is very different being in the role of consultant vs. full-time employee. At this client, we are not empowered enough to affect change as much as we would like. I think we will successfully implement Scrum, and eventually the client will be happy. We'll come out other end with some battle wounds, but it will be well worth it. Will the project succeed? I don't know. After the first sprint, the velocity isn't what executive management was expecting. Actually, not even close. But, its honest and realistic. Something they have never had.

Improving Scrum Roll-Out

I know that there can't be a fine-grained plan to implement agile development on every team. Every team will have unique needs that will have to be addressed. However, our overall approach will be different.

• We will spend a significant amount of time with the management team, getting to know their problems, their personalities, and where they want to be. Each one of them will have different perspectives that will eventually have to be addressed. We need to set expectations better than we have in the past.
  
• We will work at building trust and teaching Scrum and agile development from the top-down AND bottom-up. With our current client, we used a bottom-up approach, and it just didn't work. However, if we would have only pushed it from the top, the team members would have totally rebelled. We will have to carefully map out our approach at our next engagement.
  
• We will spend significant time educating the team on agile development and Scrum. This will have to be continual. We did a "light" introduction, and then dove right in. The team really needed more than this.
• We will make sure to not get sucked into putting out fires, which is what happened at our current client. Now, this is a tricky one. Most likely there will be fires that they will need help with. If so, we will have to introduce a separate "fire-fighting" team. While the process improvement team continues with improving the process and coaching the members of the Scrum team.
• We will need to know exactly what level of empowerment we have and/or need. While management may say that we can do whatever it takes, we will have to make sure that everything is communicated clearly and frequently. And, we will get push-back, as we've learned at this current client. We will now be more prepared.
  

We have learned a ton. Implementing Scrum as a consultant is much different than implementing Scrum at a company that you work for full time. It was very naive of me to think there would be no difference.

I will add posts as this project progresses, and as I learn new things about Scrum. I apologize for the length of this post. I promise future posts won't be like this. I do hope that this has helped someone out there who is starting their career as an agile consultant.