Last week I was meeting a friend of mine who also happens to be a software engineer and a programmer. So, as we were discussing he came around to complaining as to how his team was riddled with problems on a project that was supposed to be well planned and well organized from the start. It seems the project is now over schedule and over budget and the client is not happy. Now most of us in the software industry would just laugh that off saying, “Oh, that happens everywhere, all the time, so what else is new!” But later it left me wondering as to why this happens to all but the most trivial software projects. Why do projects that are planned and organized by people who have worked in the industry for years (, which by the way includes me), fail to deliver on time and/or on budget and sometimes do so miserably!?!! What is so difficult about estimating software development cycle that it always goes wrong; time and again. Yes there are always excuses, feature creep, attrition, inexperienced programmers, the weather; but still if you were to look at other fields of engineering like mechanical or construction you won’t see similar problems occurring there. Projects do go off schedule but the situation is much better off than what we experience in software. Why? Working in software I have seen projects go off schedule or off budget by 100% or more, worse, some end up as Vaporware.

So what’s exactly is wrong with our time/budget estimation techniques when applied to software? Well; maybe it’s not estimation techniques at all, maybe it’s our software development processes that is at fault, which in turn cause wrong time estimations. Through the years people have come up with several different software development models, but have never agreed upon a single one to be the correct one. There was, or rather is the waterfall model, which according to me falls short because it is too rigid. For most projects, changing requirements are a way of life and the waterfall model is just not geared for changing requirements. If the requirements do change, the phases of the model will overlap and thus put the whole process into complete disarray. Also the waterfall model is criticized for being too documentation oriented (requirement, design and other process documents) and focuses less on work related methodologies and productivity. There are several disadvantages of the waterfall model which would require maybe another blog entry. So I refrain myself from going too deep here. However, proponents of the waterfall model are blind to the idea that for most projects, specifications do change and there is actually very little the clients can do about it. Sometimes a spec change is a must to deal with rapidly changing customer demands, sometimes things are not clear upfront especially in domain specific projects, sometimes it just something the management doesn’t like or wants changed. Whatever the reason may be; time estimation for a non-trivial project with changing requirements using the waterfall model will be almost next to impossible. (You end up casing your own tail and doing “Adhoc – things” in the end).

OK so it maybe clear by now that I hate the waterfall model. Well I don’t. I just think the usefulness of the waterfall model is limited to projects with rigid specs. It is not something to be used everywhere and anywhere. Yes, if you read my above argument then you can see that it would clearly not fit for a broad spectrum of projects. In support of the waterfall model however; it is very easy to estimate time for a project provided the model fits to the project’s need. For a project that has unchanging specs, this model is probably the best. Now, having discounted the waterfall model leaves us with another popular model or rather a class of models called iterative models to software development. Where the waterfall model discourages change, most iterative models embrace it. They scale very well to changing requirements and accommodate spec changes rather easily. There are a lot of different iterative models and each one has it’s share of advantages and disadvantages. I don’t claim to know each and every one of them and I have only used one in my current project and that too with some custom modifications (, hybridized with the waterfall method, more on that later). What I want to focus on is the fact that though iterative models are designed to be scalable, forecasting or predicting time-lines is very difficult. If a spec change does come in, it can be easily absorbed by the process but would still ultimately end up causing disruptions in the total time estimates for the project.

While iterative methodologies look wonderful on paper, they have their own difficulties when put into practice. For example if a customer has changing requirements every week, how do you adjust your costing based on his demands every time. What do you define as a change, does changing the header of the Dialog signify as a change? Do you bill him for that? These methods are a sort of a license to the client to change anything, without strict and clear boundaries, and it often leads to all sorts of trouble. Smaller changes can slowly pile up into something involving a monstrous change in the project. While such changes can be avoided by a proactive and vigilant project manager, they have the capability to sneak up on you, particularly in an iterative approach. Also Iterative methods assume your client to be fairly technical minded or even sometimes at par with the developers working on the project. Most often they are not, and there can be huge communication gaps if that is indeed the case. In my early years as a developer I remember working on a finance project that involved graph plots for stock analysis. I spent months getting data off a file into the graph plot which was fairly extensive and at the very end of the project the client says, “Oh that’s great. Now we just need to get the same data off the live market feed instead of the file. Oh yes and we are going live on Monday.” Yes you guessed it, it was Friday then! This is clearly a case of client-team miscommunication in spite of being updated regularly. Something that looks fairly obvious to a client may turn out to be a technical nightmare for the dev team. Iterative methods work great for in-house or RnD projects where specs change, and there are intermediate deadlines but most of the team is highly technical minded. They are also excellent candidates for project managers who want to outsource their projects or a part of an existing project to some other team. Also projects that fall neatly into this category are projects that are remotely managed or projects that have members of the team spread out at physically different locations (, for example open-source projects on sourceforge).

Even if a particular project were to neatly fit into either waterfall or an iterative model or any other model for that matter, there is still no guarantee it will be completed on time and on budget. Why you might ask? The micro level reasons may be several, but according to me you could broadly classify them into 3 different categories a) Lack of clarity b) Feature creep c) Attrition. A delayed/over-budget project may sight one, two or all the three reasons for it’s eventual downfall. Attrition is something that hits projects following any model hard but generally projects following the waterfall model are the hardest hit, of-course depending on which the phase the project is in. If a programmer leaves a project mid-way the iterative models are in a better position to cope simply because the entire process is far more dynamic, however in no way are they immune to attrition. A person leaving the project also carries with him knowledge about the project and a person coming in has to reacquire at least some portion of the knowledge that was lost when the other project member left. This takes time and depending on the person coming in, his ability to cope and the complexity of the project, this could very well put a significant dent in the time required for overall project completion. But there is nothing you can do really. It is easy to say that documenting “everything” will solve the problem. Project managers often falsely think that the person coming in will only have to read up the project docs to come up to par and that too sooner rather than later, but this just doesn’t help or happen in practice. A person coming in requires a significant buffer phase and there is simply no way to predict how much time it will really take for the new person to be fully productive.

Feature creep is something that hits projects following the waterfall model the hardest and in any phase. Feature creep will just tear the model apart. If a feature needs to added midway into a cycle then the waterfall model just doesn’t have a quick answer. Iterative models are much better suited to handle feature creep but still suffer from delayed time schedules. So is it possible to avoid feature creep? The answer is somewhere between a complete yes and complete no. Too much of feature creep can result in an “infinite beta”; the product never really gets shipped. But feature creep does happen since software development is a very dynamic process. Feedbacks from beta or even alpha releases, marketing pressures and simply lack of clarity (discussed later) are all reasons for feature creep. This is something that can’t be avoided in most projects, especially game projects, something like the project I am working on. The short answer is, clients and managers should be pragmatic in their approach, but at the same time get ready for delayed schedules! Feature creep and lack of clarity are interrelated, but lack of clarity can also cause delays in features that were already planned in a project. Let me focus on first reason; that being lack of clarity due to inexperience or unclear vision. Failure on the part of the manager or the team/tech leader can result in oversight of problems or miscalculation of time estimates. The manager/tech-leader simply doesn’t have the expertise required to full understand a problem while planning or designing the project, which manifests itself later during implementation. These are the worst kinds of problems, because these problems lead to even more problems. The second reason why these problems occur is just because of the nature of software development. In software development you solve new problems all the time. Some of us more than others, and that in itself leads to a big question mark. Each software project is unique and has unique problems associated with it, problems that no one has ever solved. Sometimes estimating time for these problems is almost next to impossible. This is absolutely the reason why algorithmically intensive projects or projects that involve radical solutions end up getting delayed. No, there is simply nothing you can do about it. It is easy to say that careful and intensive planning will solve such problems. The fact however is you can’t really estimate the time required for such issues until you go down to the actual line of code, and that ironically means actually implementing it.

So that begs us to ask the question, “Is software development inherently unpredictable?”. Will we ever be able to determine the time required for software projects accurately? Yes I have seen some processes work for one project but I have also seen the same process completely fail for the other. So is the process at fault? Can it even be called a process? I refuse to call a system that can’t be replicated “a process”. Is there even such a process for accurate time prediction that is conceivable? Tough questions, but unfortunately I don’t have the answers!