With the advent of Agile techniques, thankfully companies that try to control every aspect of a software project in order to measure progress are becoming a thing of the past, thankfully far fewer than twenty or thirty years ago when I started as a software developer.
I blogged about the dangers of tight control a while ago, in January of this year, in The Dangers of Planning Time on Software Projects.
I also recommended Ron Jeffries book, The Nature of Software Development recently to people who had not discovered Agile techniques. Ron is a founding member of the Agile manifesto and this is an excellent introduction to managers and developers alike. In his book, Ron states that it is a bad idea to try to create tasks and control these within a sprint and even if you must do that, don’t specify stretch goals which are tasks that developers “get to” if they finish other planned tasks.
I was looking for a really good introduction to Agile to be able to recommend to others and this book definitely is it. Anyway on with the post. Why does tight control not work for software projects?
1. Software Development is Design, not Production and it is impossible to estimate tasks accurately, certainly anything like half a day, let alone less than that. You cannot see the details at estimation time, and if you can you might as well have done the work having far increased your time by giving such a detailed estimate.
2. If the developer is doing something that has not been done before, either for themselves or for the project, it will likely take longer and be more difficult to estimate. If it involves learning something new – a pattern or technique – the estimate could be 3 or 4 times over. This is hard to justify without looking bad in front of your manager (if your manager doesn’t understand).
Extra Overhead Involved
3. Developers are not always the best documenters so many details are very difficult to justify.
4. Documentation is required to communicate with managers both on tasks and on making copies of technical details because managers cannot read code (using techniques such as Domain Driven Design or just plain good object orientated design, modern code can be structured in such a way that the intent is visible in the code, so no need to duplicate lots of small details in Word and Excel spreadsheets, that then need to be maintained). Note: this is not an excuse for Agile projects to document or design nothing at all but what is documented must be properly maintained along with the codebase.
5. It is not always possible to explain and justify detailed concepts to a manager who was a COBOL programmer once but doesn’t really understand that he/she doesn’t understand modern practices and patterns. Result: either a lot of time spent trying to justify, or the detail might just get missed, or a simpler to justify technique (that is worse for the project) might be used instead. Maybe you just start coding and hope for the best…
6. There is a risk that tasks will be specified from above in an inefficient or impossible to achieve order, which dictates creating software components earlier in the development process than otherwise would be the case. Agile techniques leave detailed decisions until they need to be made. They are not over specified up front. Also concentrating on Agile’s value proposition and continuous delivery helps mitigate this problem in any case. I once worked on a big project like this with nothing whatsoever to deliver until the end – the “big bang” approach which is one of the reasons software projects fail – the requirements move on so much during the time the software is being developed, its unusable when delivered.
7. If an overrun occurs, the developer is then full time on the extra overhead of trying to justify why the estimate was wrong and updating the task database when the developer should be concentrating 100% on delivering the correct solution for the software rather than breaking a chain of thought that led to that point.
8. If a concept or technique is developed that isn’t 100% correct first time round, then extra time will be required. Does the developer soldier on and keep their head down, or go to their manager and ask for more time and incur a lot of justification, admin and look bad in front of their manager? Or do they ignore and just complete the task the best they can thinking they can come back later? Agile techniques practice continual improvement in response to continual change and learning of requirements from customers. The developer is continually re-factoring and improving code. The non agile manager could see this re-factoring as paying for the job twice or worse when the opposite is the case.
9. The risk of technical debt if there is too much interference (from any of these points), or if there is nobody in the team to shield the developer from excess pressure. All teams need senior developers in my view, to help shield excess pressure from above, to help with strategy and to guide and develop other developers skills in a way that is beneficial and doesn’t impact projects too much.
Trying to Monitor Performance
10. Monitoring performance is a management problem, that management should ideally carry out with a minimum of interference on the work involved. Many in the NHS or schools have this problem. Because the politicians need “facts” so they can cherry pick and obfuscate them for their own ends, they double the overhead in these places just because they need figures they are going to change anyway. Who wants to work for a job with the prime minister looking over your shoulder telling you how to do your job? Thankfully in software, we have Agile techniques that allow us to avoid this problem.
11. The risk of constraining the project to a size and complexity that a manager can understand all the details or to what can be justified to managers. Even though Larry Page and Sergey Bryn are good programmers, at Google engineers stopped both from programming early on in Google’s history, possibly for this reason, I don’t know for sure I wasn’t there.
12. Sometimes there are times when the team knows what do do. Management (team leaders included) should just get out of the way and the developers should get on the best they know how without hindrance.
13. Finally education can suffer. Development by its very nature is a learning process where the team learn about both the application they are producing and the best techniques to use as they go along. Not allowing time or budget for education, especially if this impacts your software architecture is a really bad thing. Also these kind of companies can’t justify 20% time such as that practised at Google, where 20% of the time developers are free to develop anything they want – this will be normally things that better their own skills and the companies fortunes but which they can’t justify in advance to a manager. The classic example of this was Adsense at Google but I am sure there are many among the enlightened companies that allow 20% time or who have good developer training budgets.
14. Often projects like this will overlook architecture considerations, correct object orientated design, or domain driven design, because the managers controlling don’t really know what they are, how they relate to a project, or why they are so important and useful to ensuring the absolutely best product is delivered in the shortest possible time. Result: Work has to be done many times over and due to the lack of architecture contains many bugs rather than being bug free first time, and is never right, or anywhere near as useful to the end user as it should be.
All or any of the above can double or triple the size (time taken) of a software development project. Tight control is really unnecessary in this day and age. Not having a route to channel concerns or make those concerns hard to justify, can double or treble it again, as can not having a suitable developer or manager to shoulder and handle excess pressure from management. Excess pressure leads to code smells and technical debt. This is without considering the actual practices and patterns used when developing the software.
I have worked on some projects where even getting a web project to use caching has been a mammoth task and has taken a couple of days writing detailed documents and then a couple of months to schedule a meeting to discuss the document just to try and justify it. A couple of days is enough to develop and refine the caching algorithm to use and trial it in the project to see what actually works for the system and letting the results speak for themselves. That is the point of Agile – don’t try to justify anything, let the software speak for itself and if it isn’t right, change and add to it until it is.
How to avoid these pitfalls? Follow Agile techniques. Measure what is actually planned and delivered for each sprint, and leave the developers to get on with each sprint without hindrance or slowing them down by asking them to do admin tasks and justify being there all the time. Control the project by changing priorities based on value delivered for future sprints. Do not mess with current sprints where at all possible. Do not over specify to the nth degree and then interfere with the detail as well as the time being spent. Specify all requirements in user speak. Do not try to tell the development team how to do things and when to do them apart from specifying requirements and sprints to do them in.It really is a recipe for disaster.
Ron Jeffries book is on my Software Architecture and Design Books page. Highly recommended.
Another way of looking at all of this is to be Agile and measure on results – i.e. on deliverables delivered within timescale and budget, not on specific detailed data. If your project is not delivering, ask yourself why. You can also contact me and ask me why if you like. If you are open to new ideas, I’ll come along and help you sort it all out – to your complete satisfaction – no problem at all.