Why Big Projects Fail

Recently some coworkers of mine mentioned that they were reading The Mythical Man Month in a reading group. It really should be required reading for anyone who wants to lead a project of any size. It’s a relatively short read, but the lesson in it is extremely important. It was one of the first books I ever read when I was at MIT, and was actually part of one of the first classes on software development I took. I still take the lessons from that book very seriously, and it is one of the reasons that I believe working Agile will in the long run be better for all of us. We are all aware that there is a point when it doesn’t matter how many people you throw onto a project or even how well you document something that is really complex, it just hits a point of diminishing returns or even ends up making the process take longer than it would with less people.

One of the classes that I took at MIT was called Computer Systems Engineering. Oddly enough, it was a class that involved a lot of reading and writing. We would read about all sorts of technologies like Kerberos, Operating Systems, Threading, etc., and then we would be tasked with writing essays on these various topics. One of the lectures was all about big engineering projects that failed. They started out small and talked about projects that companies had put in place to upgrade or replace dated technologies. As the lecture went on, the amount of money and resources that went into each project continued to increase. Near the end of the lecture, they talked about how British Airways started a project to replace their entire reservation system. The company worked on the project for nearly two years, spent almost a billion dollars (yes, with a b) and in the end, had to scrap the entire project.

The moral of the lecture was pretty simple: complexity kills projects. Imagine you were trying to build the Space Shuttle from scratch. That is the kind of thing where you have to design every detail very carefully and rigorously test it over and over because if you are wrong, people die. If you have ever seen the move The Right Stuff, you know that America’s Space Program was only successful after years and years of failed tests. Once they even pushed the button to launch a rocket and the only thing that happened was the top part of the capsule that holds the parachute popped off. After years and years of failures, they finally had systems that worked, and eventually they worked so well that the number of casualties has been extremely small considering how dangerous Space really is. The complexity of it, however, made it take years and years. As engineers, we don’t have years and years. We can’t constantly build things, test them, and watch them blow up.

Instead, we need to break those really complex things down into smaller pieces. We can move forward if we define the problems as smaller tangible problems that can be easily defined, easily implemented, easily tested, and easily delivered. The goal isn’t to deliver the whole system at once, but rather to build a piece at a time (or many pieces in parallel – think of how the ISS space station was built, with many countries building small pieces that fit together to make the whole). This is why the Microservices approach is really taking the world by storm, because quite simply, it is easier to deliver a thousand small things that are independently testable than to try to deliver one big thing that requires thousands of tests.

One problem that I often witness is that companies often struggle just defining what it is that they want to build. Someone defines a grand vision of the direction the company wants to take. “We want to better connect with our customers and deliver what they want, not what we think they want.” The vision is sound, but how does it translate into actual business requirements? Unfortunately, a lot of us are stuck with some of the architectures we have, and so we define it in terms of how to get where we are to where we want to be. Therefore, we start drawing a big architecture that we dream is the future. As we begin working on it, however, we realize that the requirements for the big architecture are complex and not well defined, and the project stalls while we work out what those actual requirements are and what approach will be the best to meet those requirements.

To put it simply, the problem you are trying to solve is actually many problems. It is just as if you were trying to figure out how to clean your whole house at once. There are so many different things that need to be done that you could get lost in trying to figure out what you should do first or how to best accomplish the task. So how do you tackle this big problem? You break it down into smaller tasks. Clean the kitchen. Clean the bathrooms. Clean the bedrooms.

Each of these tasks have their own subtasks. Clean the fridge. Clean the toilet. Pick up clothes on the floor. Now you have simple, tangible tasks where the requirement is clear and the task can be completed in a short measurable period of time. Not only that, you can pick and choose which tasks as you go – you might get tired of cleaning bathrooms and decide to work on the kitchen instead. You might realize you have friends coming over later, so you switch to cleaning the living room first. You can change direction at almost any time and come back to other tasks later.

This is what Agile really is about. If you don’t break your project down into small tangible pieces that your engineering team can clearly understand, how can you really expect them to deliver what you actually want? How can you expect that you even know what you really want? When are you actually ready to start? It may give some people comfort believing that everything has been planned or designed ahead of time, but how many times have you ever seen a project that went exactly according to plan? Never. Not once. Requirements change. People leave. Budgets get cut. Timelines change. If you keep trying to deliver software in a way that can’t adapt to change, you will fail.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s