Why do Software Projects Fail?

First of all, what is considered a failure? Shipping late, without needed features and with considerable bugs would be considered failures by most people. Shipping a product that has no bugs but doesn't meet anyone's needs certainly cannot be considered a success, and shipping "on time" but without a feature set that customers want could only be considered a failure.

So, why do software projects fail? Having participated in several software projects over the past 20 years, both successes and failures, I have come to the conclusion that failure is actually fairly easy to spot, and actually not that difficult to fix.

Some of the main reasons for failure are:

1. No Clear understanding of customer needs. The customer representatives in the project need to have a very clear understanding of customer needs - which the customer themselves may not always be able to clearly articulate. Henry Ford once said "If I had asked my customers what they wanted, they would have said 'Faster Horses' ". To provide a truly innovative solution that customers will love, you need to understand their needs and fill them rather than simply implementing the features that customers request. In the absence of a clear understanding of customer needs, many teams fall back to "Self-Referential design". This is very dangerous, as it leads to development of applications that are very well suited to the needs of the development and marketing teams that created them. Most of the time, this only partially meets the needs of the target customer.
   
2. No Clear Owner. All too often, software product development is divided between a marketing-owned group who identifies features and prioritizes bug lists, and an engineering team that implements the code. The leaders of the engineering and marketing groups are supposed to "co-own" the product, and supposedly there should be some "healthy friction" between them. This thought process is flawed, and only leads to "Design by Committee". Alan Cooper said: "When everyone has responsibility for Product Quality, nobody is responsible for Product Quality". I have never seen this type of process to be successful. Donald Norman, the author of "The Design of Everyday Things", has been quoted as saying: "You don't do good software design by committee. You do it best by having a dictator. From the user's point of view, you must have a coherent design philosophy... The person who's done it best is Steve Jobs, and he's well-known for being a tyrant."
   
3. Lack of Discipline. You need to know when the "train has left the station" (which, incidentally, is NOT when the product has shipped, but rather when the time for adding more features has passed), and to make the tough decisions regarding which bugs and enhancements to postpone and which to include in a release. Everyone wants the project to be successful. The problem is that "success" is not clearly defined at the outset, so everyone has a different view of what success means. Because of this, a gap exists between members of the team, and it becomes very difficult to draw the line of what will be fixed and working, and what may need to ship as a "known issue". Without this clear understanding, it is very difficult to determine when the "train has left the station", and scope-creep (or "feature-creep") ensues. It takes strict discipline on the part of development, quality assurance and product management to ensure that quality products are shipped on-time.
   
4. Putting too much on the plate. At the start of a project, some lines must be drawn. The entire team must understand what is needed, and when the product is fully-baked. Too often project teams put too much on the plate at the start of the project, and set themselves up for failure. The key is to define very limited-scope iterations that can be accomplished in a short time period. Everyone needs to understand what the goal is of this iteration, as well as what the over-arching goal is of the entire project (i.e., the company "mission"). Breaking down the overall goal into mini-goals, and defining short iterations helps to keep projects on-track, and ensures high quality as the project progresses.
   
5. Silos. Development needs to operate in a fishbowl, and there must be complete transparency into what they are working on. The development organization must have "windows" into their daily operations, and those outside of development need to be willing to look through the window. Whenever a failed project ships, I inevitable hear that those outside of development didn't know what the developers were working on until it was too late. There are two possible reasons for this:
1. Development is operating in a silo. Many development managers feel that in order to maintain discipline and remove feature-creep, they need to keep the development team insulated from the outside influences of marketing and sales. While this is quite common, it is dead wrong.
2. The windows do indeed exist into the strange world of development, but those in other departments don't take the time to look through them. Development may indeed be operating in a very transparent model, but if marketing, sales and others don't take the time to look through the window, they will always be surprised at the end of the project.
   
6. Developers are not Normal People. Managing software developers is counter-intuitive to most non-technical managers. The motivation is different, and most standard management techniques do not work for these "artistic scientists". Too often, when upper-management feels a project is getting off-track, they get more involved, and begin to take over the project by applying standard tried-and-true management techniques from the world of Sales, Marketing and other groups of "normal" people. These attempts to apply standard management techniques that would work with sales, marketing, customer support or other job functions, usually backfire and lead to a downward spiral of more failed projects, more non-developer management techniques to solve it, etc.

These are all fairly standard pitfalls, and I have seen them in many different companies over the years. The solution to these problems is quite simple, but requires a software development culture shift in order to make it work. In future posts, I will attempt to describe each of these in more detail, and provide guidance on how to create a software culture that removes these obstacles from your software development environment, which leads to projects that are positioned for success.