Last week I was meeting a friend of mine who also happens to be a software engineer and a programmer. So, as we were discussing he came around to complaining as to how his team was riddled with problems on a project that was supposed to be well planned and well organized from the start. It seems the project is now over schedule and over budget and the client is not happy. Now most of us in the software industry would just laugh that off saying, “Oh, that happens everywhere, all the time, so what else is new!” But later it left me wondering as to why this happens to all but the most trivial software projects. Why do projects that are planned and organized by people who have worked in the industry for years (, which by the way includes me), fail to deliver on time and/or on budget and sometimes do so miserably!?!! What is so difficult about estimating software development cycle that it always goes wrong; time and again. Yes there are always excuses, feature creep, attrition, inexperienced programmers, the weather; but still if you were to look at other fields of engineering like mechanical or construction you won’t see similar problems occurring there. Projects do go off schedule but the situation is much better off than what we experience in software. Why? Working in software I have seen projects go off schedule or off budget by 100% or more, worse, some end up as Vaporware.
So what’s exactly is wrong with our time/budget estimation techniques when applied to software? Well; maybe it’s not estimation techniques at all, maybe it’s our software development processes that is at fault, which in turn cause wrong time estimations. Through the years people have come up with several different software development models, but have never agreed upon a single one to be the correct one. There was, or rather is the waterfall model, which according to me falls short because it is too rigid. For most projects, changing requirements are a way of life and the waterfall model is just not geared for changing requirements. If the requirements do change, the phases of the model will overlap and thus put the whole process into complete disarray. Also the waterfall model is criticized for being too documentation oriented (requirement, design and other process documents) and focuses less on work related methodologies and productivity. There are several disadvantages of the waterfall model which would require maybe another blog entry. So I refrain myself from going too deep here. However, proponents of the waterfall model are blind to the idea that for most projects, specifications do change and there is actually very little the clients can do about it. Sometimes a spec change is a must to deal with rapidly changing customer demands, sometimes things are not clear upfront especially in domain specific projects, sometimes it just something the management doesn’t like or wants changed. Whatever the reason may be; time estimation for a non-trivial project with changing requirements using the waterfall model will be almost next to impossible. (You end up casing your own tail and doing “Adhoc – things” in the end).
OK so it maybe clear by now that I hate the waterfall model. Well I don’t. I just think the usefulness of the waterfall model is limited to projects with rigid specs. It is not something to be used everywhere and anywhere. Yes, if you read my above argument then you can see that it would clearly not fit for a broad spectrum of projects. In support of the waterfall model however; it is very easy to estimate time for a project provided the model fits to the project’s need. For a project that has unchanging specs, this model is probably the best. Now, having discounted the waterfall model leaves us with another popular model or rather a class of models called iterative models to software development. Where the waterfall model discourages change, most iterative models embrace it. They scale very well to changing requirements and accommodate spec changes rather easily. There are a lot of different iterative models and each one has it’s share of advantages and disadvantages. I don’t claim to know each and every one of them and I have only used one in my current project and that too with some custom modifications (, hybridized with the waterfall method, more on that later). What I want to focus on is the fact that though iterative models are designed to be scalable, forecasting or predicting time-lines is very difficult. If a spec change does come in, it can be easily absorbed by the process but would still ultimately end up causing disruptions in the total time estimates for the project. Continue reading