A Blog about Coding

Category: Agile Techniques

  • 14 reasons why Tight Control of Software Projects doesn’t Work

    14 reasons why Tight Control of Software Projects doesn’t Work

    With the advent of Agile techniques, thankfully companies that try to control every aspect of a software project in order to measure progress are becoming a thing of the past, thankfully far fewer than twenty or thirty years ago when I started as a software developer.

    I blogged about the dangers of tight control a while ago, in January of this year, in The Dangers of Planning Time on Software Projects.

    I also recommended Ron Jeffries book, The Nature of Software Development recently to people who had not discovered Agile techniques. Ron is a founding member of the Agile manifesto and this is an excellent introduction to managers and developers alike. In his book, Ron states that it is a bad idea to try to create tasks and control these within a sprint and even if you must do that, don’t specify stretch goals which are tasks that developers “get to” if they finish other planned tasks.

    I was looking for a really good introduction to Agile to be able to recommend to others and this book definitely is it. Anyway on with the post. Why does tight control not work for software projects?

    Estimation

    1. Software Development is Design, not Production and it is impossible to estimate tasks accurately, certainly anything like half a day, let alone less than that. You cannot see the details at estimation time, and if you can you might as well have done the work having far increased your time by giving such a detailed estimate.
    2. If the developer is doing something that has not been done before, either for themselves or for the project, it will likely take longer and be more difficult to estimate. If it involves learning something new – a pattern or technique – the estimate could be 3 or 4 times over. This is hard to justify without looking bad in front of your manager (if your manager doesn’t understand).

    Extra Overhead Involved

    3. Developers are not always the best documenters so many details are very difficult to justify.
    4. Documentation is required to communicate with managers both on tasks and on making copies of technical details because managers cannot read code (using techniques such as Domain Driven Design or just plain good object orientated design, modern code can be structured in such a way that the intent is visible in the code, so no need to duplicate lots of small details in Word and Excel spreadsheets, that then need to be maintained). Note: this is not an excuse for Agile projects to document or design nothing at all but what is documented must be properly maintained along with the codebase.
    5. It is not always possible to explain and justify detailed concepts to a manager who was a COBOL programmer once but doesn’t really understand that he/she doesn’t understand modern practices and patterns. Result: either a lot of time spent trying to justify, or the detail might just get missed, or a simpler to justify technique (that is worse for the project) might be used instead. Maybe you just start coding and hope for the best…
    6. There is a risk that tasks will be specified from above in an inefficient or impossible to achieve order, which dictates creating software components earlier in the development process than otherwise would be the case. Agile techniques leave detailed decisions until they need to be made. They are not over specified up front. Also concentrating on Agile’s value proposition and continuous delivery helps mitigate this problem in any case. I once worked on a big project like this with nothing whatsoever to deliver until the end – the “big bang” approach which is one of the reasons software projects fail – the requirements move on so much during the time the software is being developed, its unusable when delivered.

    Overruns

    7. If an overrun occurs, the developer is then full time on the extra overhead of trying to justify why the estimate was wrong and updating the task database when the developer should be concentrating 100% on delivering the correct solution for the software rather than breaking a chain of thought that led to that point.
    8. If a concept or technique is developed that isn’t 100% correct first time round, then extra time will be required. Does the developer soldier on and keep their head down, or go to their manager and ask for more time and incur a lot of justification, admin and look bad in front of their manager? Or do they ignore and just complete the task the best they can thinking they can come back later? Agile techniques practice continual improvement in response to continual change and learning of requirements from customers. The developer is continually re-factoring and improving code. The non agile manager could see this re-factoring as paying for the job twice or worse when the opposite is the case.

    Technical Debt

    9. The risk of technical debt if there is too much interference (from any of these points), or if there is nobody in the team to shield the developer from excess pressure. All teams need senior developers in my view, to help shield excess pressure from above, to help with strategy and to guide and develop other developers skills in a way that is beneficial and doesn’t impact projects too much.

    Trying to Monitor Performance

    10. Monitoring performance is a management problem, that management should ideally carry out with a minimum of interference on the work involved. Many in the NHS or schools have this problem. Because the politicians need “facts” so they can cherry pick and obfuscate them for their own ends, they double the overhead in these places just because they need figures they are going to change anyway. Who wants to work for a job with the prime minister looking over your shoulder telling you how to do your job? Thankfully in software, we have Agile techniques that allow us to avoid this problem.

    Other

    11. The risk of constraining the project to a size and complexity that a manager can understand all the details or to what can be justified to managers. Even though Larry Page and Sergey Bryn are good programmers, at Google engineers stopped both from programming early on in Google’s history, possibly for this reason, I don’t know for sure I wasn’t there.
    12. Sometimes there are times when the team knows what do do. Management (team leaders included) should just get out of the way and the developers should get on the best they know how without hindrance.
    13. Finally education can suffer. Development by its very nature is a learning process where the team learn about both the application they are producing and the best techniques to use as they go along. Not allowing time or budget for education, especially if this impacts your software architecture is a really bad thing. Also these kind of companies can’t justify 20% time such as that practised at Google, where 20% of the time developers are free to develop anything they want – this will be normally things that better their own skills and the companies fortunes but which they can’t justify in advance to a manager. The classic example of this was Adsense at Google but I am sure there are many among the enlightened companies that allow 20% time or who have good developer training budgets.
    14. Often projects like this will overlook architecture considerations, correct object orientated design, or domain driven design, because the managers controlling don’t really know what they are, how they relate to a project, or why they are so important and useful to ensuring the absolutely best product is delivered in the shortest possible time. Result: Work has to be done many times over and due to the lack of architecture contains many bugs rather than being bug free first time, and is never right, or anywhere near as useful to the end user as it should be.

    All or any of the above can double or triple the size (time taken) of a software development project. Tight control is really unnecessary in this day and age. Not having a route to channel concerns or make those concerns hard to justify, can double or treble it again, as can not having a suitable developer or manager to shoulder and handle excess pressure from management. Excess pressure leads to code smells and technical debt. This is without considering the actual practices and patterns used when developing the software.

    I have worked on some projects where even getting a web project to use caching has been a mammoth task and has taken a couple of days writing detailed documents and then a couple of months to schedule a meeting to discuss the document just to try and justify it. A couple of days is enough to develop and refine the caching algorithm to use and trial it in the project to see what actually works for the system and letting the results speak for themselves. That is the point of Agile – don’t try to justify anything, let the software speak for itself and if it isn’t right, change and add to it until it is.

    How to avoid these pitfalls? Follow Agile techniques. Measure what is actually planned and delivered for each sprint, and leave the developers to get on with each sprint without hindrance or slowing them down by asking them to do admin tasks and justify being there all the time. Control the project by changing priorities based on value delivered for future sprints. Do not mess with current sprints where at all possible. Do not over specify to the nth degree and then interfere with the detail as well as the time being spent. Specify all requirements in user speak. Do not try to tell the development team how to do things and when to do them apart from specifying requirements and sprints to do them in.It really is a recipe for disaster.

    Ron Jeffries book is on my Software Architecture and Design Books page. Highly recommended.

    Another way of looking at all of this is to be Agile and measure on results – i.e. on deliverables delivered within timescale and budget, not on specific detailed data. If your project is not delivering, ask yourself why. You can also contact me and ask me why if you like. If you are open to new ideas, I’ll come along and help you sort it all out – to your complete satisfaction – no problem at all.

  • Technical Debt isn’t a Technical Problem

    Technical Debt isn’t a Technical Problem

    I listened to an old podcast from Dot Net Rocks recently Technical Debt isn’t Technical with Einer Host, which went on to explain how Technical Debt isn’t a technical problem.

    Using Agile techniques, once the sprint has been set by the product owner, programmers are left to themselves to organise the work and to slowly build the software feature by feature, while keeping a fully tested releasable version of the software available at the end of every sprint.

    Wikipedia (https://en.wikipedia.org/wiki/Technical_debt) says:

    Technical debt (also known as design debt or code debt) is a metaphor referring to the eventual consequences of any system design, software architecture or software development within a codebase. The debt can be thought of as work that needs to be done before a particular job can be considered complete or proper”

    Some technical debt is inevitable due to the nature of the very iterative, simplest method possible way of programming in Agile, but it is constantly re-factored out and improved upon as the work progresses. If it is not, it means the work done so far is probably adequate and the debt is a very small thing that works but possibly is not best practice, but is such a small thing it doesn’t really matter or affect anything.

    There is another side to technical debt however, that is if the team is not really Agile, and there are problems with management interfering or trying to control work, or by trying to set deadlines rather than measuring velocity, then an amount of technical debt that can build up that eventually slows the project down to the point where further development work is very difficult, expensive and time consuming. This problem is as bad as not using good design in the first place due to having a very inexperienced team doing the work.

    The Wikipedia article lists many more reasons for technical debt and the pod-cast was a very interesting listen. The common causes listed in the Wikipedia article all fall into two categories:

    1. Management Pressure to try to push or control things resulting in things being done in the wrong order or not at all
    2. Inexperience on the part of Developers

    The answer? In my view, try to follow Agile practices as far as possible, because the more a project does something that is new, the more a project can benefit from Agile techniques. Agile techniques can handle uncertainty and change better than any other method we know. I also believe that seniors should work to support the whole team in terms of knowledge and ensuring they have enough time to carry out their work and also shield the team from management pressure (although not allowing the current sprint to be altered after it has started except by developers themselves goes a long way to doing that).

  • Getting Started with Software Architecture in C# or VB.NET (Part 2)

    Getting Started with Software Architecture in C# or VB.NET (Part 2)

    This post is part 2 of an introduction to Software Architecture in C# or VB.NET. Part 1 covered some concepts, this post goes into a little more detail about programming by interface and SOLID principles by Uncle Bob Martin.

    Programming by Interface

    An interface is another layer of abstraction. Why bother? I’ve heard this from many people, especially people who are more inclined to look at the existing system to work out what to change, rather than look to a design of how things should be. I blogged about this a while ago.

    Programming by Interface is an important concept whether you are interested in TDD or not, because it allows you to control (by explicitly defining) how different modules link together, i.e. dependencies. It also allows you to plug in TDD on anything you see fit, e.g. complex business logic that is absolutely fundamental to the working of your system.

    Interfaces can be inherited just like classes can. It’s the interfaces that define functionality and dependencies, the concrete classes just come in to fulfil the needs of an interface at runtime.

    This is where Dependancy Injection (DI) is typically used, just to plug in concrete classes when needed – for a certain part of the design. So you use new statements (or factories) in your design for certain classes, and DI instead of new statements of another set of classes. E.g. Model’s, Forms and Reports are not injected. Objects that work on the model typically are.

    Interface vs Abstract Classes

    These are more or less the same thing but slightly different. Abstract classes can provide some functionality (an interface can provide none), and also have a parent, so only single inheritance is allowed. Abstract classes should only really be used for “IS A” not “HAS A” relationships.

    Programming by Interface is the most important concept I will blog about in this overview. You should try to learn as much as possible about this because interfaces explicitly control the relationships, i.e. dependencies, between objects. They are very important, not just for those interested in TDD. To find out more, apart from search for terms like “Programming by Interface” search for terms like “Composition over Inheritence”. Wikipedia has an example of Composition over Inheritence.

    SOLID Principles

    Uncle Bob Martin came up with the SOLDI principles which then were modified to SOLID during an interview with Scott Hanselman on Hanselminutes a while ago.

    I am not going to try to define them again – a link to Uncle Bob at the end of the post does that – but I would like to comment on each of the five main principles:

    S – Single Responsibility Principle

    Classes and methods should do just “One” thing.
    The “One” is a matter of judgement viewing all your code in context. It doesn’t mean that each method can only be one line of code!!
    You end up with many small classes and methods, which actually are easy to name because although you have a lots of them, because they only do one thing, they are easy to name. Class, methods, variable names should be descriptive (no need for comments).
    Also cross cutting concerns can be implemented via helper methods (one place where statics are used) and they can be put in to certain classes (or injected in via DI or PostSharp).

    O – Open / Closed Principle

    Classes should be open for extension, closed for modification. So inheritance is always needed. One of the key OOD principles is to encapsulate what changes. So when decided what to put in a class and what can be virtual, this is a good guideline.

    L – Liskov Substitution Principle

    When using a class, you should be able to substitute any other sibling into that code without breaking it. The rectangle/square problem is a good one – I’m not going to go through it all here – but its down to should a rectangle inherit from a square (or vice versa). The answer – not it should not. Because they are different shapes and it would also violate this principle.

    I – Interface Segregation Principle

    Clients should not be forced to depend upon interfaces that they do not use (Clients being any consumer of an object, i.e. another object). This is about the structure of the interfaces between classes, keeping them small. This does not mean that you alter the layers in a layered system (see Microsoft Application Architecture Guidelines). For example, a DAL could / should? Have lots of small interfaces but containing say all the columns in a given table, the Business Layer could take advantage of whatever it needs without having to change the DAL to add extra columns etc. that are already in the table, but the façade layer that delivers the data to the UI (via WCF or otherwise) could have bigger, amalgamated classes that contain all the data needed in that transaction so only one call is required, specifically for that interaction.

    D – Dependency Inversion Principle

    High level modules should not depend upon low level modules. Both should depend upon abstractions. Abstractions should not depend upon details. Details should depend upon abstractions. This is ultimately about having the references in your visual studio project pointing the “Right” way, e.g. if you have circular references you are probably missing a layer of abstraction.

    http://www.davesquared.net/2009/01/introduction-to-solid-principles-of-oo.html is one of the best intro’s I’ve found on the web.

    The book Agile Principles, Patterns and Practices explains SOLID far better than I can – see below.

    Whats Next

    Now that I’ve summarised what I think are important concepts, in future posts I’ll go into more detail as I come across examples etc.

    Further Information – Links

    SOLID

    http://butunclebob.com/ArticleS.UncleBob.PrinciplesOfOod – definitive guide from Uncle Bob’s Blog

    http://www.davesquared.net/2009/01/introduction-to-solid-principles-of-oo.html – A good introduction

    Microsoft Application Architecture Guidelines

    A very good read

    https://msdn.microsoft.com/en-us/library/ff650706.aspx

    More about Dependency Injection (DI)

    (I don’t recommend Unity because its slow but it’s a good tutorial):

    https://msdn.microsoft.com/en-us/library/dn223671(v=pandp.30).aspx

    More Links

    www.hanselminutes.com

    www.dotnetrocks.com

    http://www.dnrtv.com/

    A couple of interesting shows from Dot Net Rocks:

    https://www.dotnetrocks.com/?show=1235 – very interesting bit about technical debt

    https://www.dotnetrocks.com/?show=727 – Steve McConnell – author of Code Complete and www.construx.com

    Uncle Bobs videos are very good and are available at a price on https://cleancoders.com/.

    There may be a few examples on his site that are free – and he is not on Pluralsight unfortunately.

    However searching for SOLID on YouTube is another possibility.

    Further Reading – Books

    If you are going to read just one book, then Head FIrst Object Orientated Design is the one followed by Agile Principles, Patterns and Practices in C#

    The books are listed in approximately the order you should read them if you haven’t read anything on the subject.

    Beginners

    Intermediate

    Advanced

    This post has not mentioned anything about the core C# programming skills that every C# programmer should have – see my page on C# programming books for further information.

  • Getting Started with Software Architecture in C# or VB.NET (Part 1)

    Getting Started with Software Architecture in C# or VB.NET (Part 1)

    Sixteen years ago I was working on a VB6 Access VBA contract. The first version of .NET had yet to be released. About that time Microsoft brought out a version of Java which was not widely accepted due to it being different to standard Java (and developers not wanting Microsoft to control Java), which led I think to the birth of C#. How times have changed since then. Despite coming up with many innovations of its own, Microsoft has been playing catch up big style in the past sixteen years in terms of software architecture. Many ideas have come over from the Java world into .NET and C#, many of them really good and which have moved the design of .NET software for the better I think.

    In this series of posts I would like to introduce some of these as if you have never heard of them before, and then in future posts I’ll delve into more detail. I hope this is the first post of eventually what might be an interesting series on software architecture and at least I will have a good resource to show people when they ask about it.

    I would like to add that despite putting as much time as I can spare into personal development, this is such a big area and there are so many new things coming out, that its impossible to be an expert in everything even if you devoted your entire working life to it. I have found for many years, like my wife, who is a scientist in a totally different area, that the more you learn the more you release how little you know. I think that’s definitely the case here. That is no reason not to learn about it though, because even just one little thing can change the software you write for the better.

    Agile

    Everybody seems to be talking about Agile as though its a new thing, but its going on 20 years old. The Agile Manifesto was written in 2001. There are various flavours of Agile techniques, but all I would like to say in this post is that one of the founders of the Agile Manifesto has written a book that is a really easy read that gives a really good overview for managers and developers alike.

    Please see the page on Books about Software Architecture and Design on this site for further information.

    Object Orientated Design (OOD)

    Pure OOD can be quite abstract to the needs of a form or report you are writing. However there are big benefits in terms of the code that needn’t make things more complex. In fact good architectural design can mean the code in more readable and the intent more visible, with much less code, so its easier to spot bugs before the code gets released. OOD is a big part of this. One of the benefits is have more but far simpler and shorter classes and methods. Another is that many design patterns will separate out functional code from plumbing type code that while essential for the code to work in the real world, is code that is muddies the picture with extra complexities and dependencies when trying to read what the code does.

    If you are at the level where you know what a class is, properties, and just a little of the theory about inheritence etc because you have used Windows Forms or Web Forms, then there is a lot of good things to learn. I can’t recommend the following book any higher:

    Domain Driven Design (DDD)

    Eric Evans brought out what is now a classic book, Domain Driven Design, he might have even coined the term (can’t remember). The whole point of DDD apart from to have a common language to discuss a system with, is to have the business domain at the centre of any system and it should be produced without any dependencies on other layers such as UI and database, in fact other layers should depend on it (i.e. the references should point inwards). This can result in the business logic for a system all in one place, in pure C# or VB.NET classes and methods, which has to be far better than having it splattered all around a system and all its objects to the point where its difficult to find all the business logic, let alone see what it does.

    This book is a very interesting read for managers and developers alike:

    Extreme Programming (XP)

    At first glance, Extreme Programming (XP) has dozens of rules that are impossible to memorise or understand without a lot of study. The reason why I’m mentioning it here is that it may be possible to pull some of the techniques out of it to use, with no need to implement fully (or have a gradual implementation). Agile and XP are the two places to look when in need of inspiration, Agile has been going 20 years and XP 15 (2001 I think it was) and have been continuously developed and improved during the time. Who am I to think I have a better way? At least I should look and see if any part of it helps me, or can be improved upon without re-inventing the wheel from scratch.

    Also, when looking to leave the campsite tidier than you found it (i.e. code in a better condition than when you found it), how do you know what “better” is without having some guidelines? Thats one of the objectives of these posts – to try to give pointers to where you can find this information, without linking to specific articles that may be out of date in no time.

    I think the following book might be out of print but its still available on Amazon as I write:

    My favourite XP “thing” – the concept of a spike – where you take a very small part of a system functionality and implement it fully – so all the different angles, database access, ui, localization, everything can exist just for one form. You might end up with loads of stubs and comments but that’s the whole point – to work out the overall architecture of the code and where everything goes ahead of time so you can work in the right direction from then, rather than having to come back and re-factor.

    Next Post in this Series

    In the next post I will go into some more detail about important architectural concepts and include more links for information for the reader.

    Summary of Resources

    Agile Manifesto
    This Site: Books about Software Architecture and Design

  • Do Agile Software Projects need a Team Leader?

    Do Agile Software Projects need a Team Leader?

    I have only worked on a few projects that have been formally Agile based on Scrum/Kanban, although have been working like this informally since about 1996.

    I have worked on many projects as a team leader or as the sole developer or designer.

    I am very interested to try to tie my own views on team leaders and documentation to the XP/Agile world. I wonder if Agile only really works when you are dealing with certain teams of experienced programmers. I think there is still a place for team leads and also that no documentation, design or planning because we are “Agile” is most definitely a bad thing. I also think that TDD should be used with intelligence, because to put under test every single line of code no matter what – because NCrunch says so, is massively increasing the amount of work the team has to do and the complexity of the software for diminishing benefit.

    Team Leader

    The purpose of a team leader in the fast moving software industry is to increase the knowledge and capabilities of his/her team even if the team contains contractors (I’ve worked at some places that specifically exclude contractors from functional discussions that they really need to know about). That way team members are the most effective possible and can do the best work in the shortest time.

    As a team leader, there are three modes of operation you can be in depending on the experience of the team and the time available to do the work:

    1. Survival Mode. This is where there is insufficient time to do anything properly, all the time available is taken by the team fighting fires.
    2. Learning Mode. There is sufficient time available to plan things and for work to be done properly, because members have the time to do it properly.
    3. Self Organisation Mode. All team members are sufficiently capable and interchangeable with each other to be able to manage themselves. This is kind of where I think Agile projects need to be, but technology changes, pressures come from the business, people leave and join the team. Its not always possible in my view.

    In my experience, not all team members are sufficiently experienced, or know enough about architecture to be able to self manage all the time. If TDD is being used, they will probably spend more time on 100% coverage (trying to TDD EF for example) rather than concentrating on the core functionality of the app they are trying to build.

    A team leader is needed to help make decisions, allocate work to different team members, help develop team members, and finally get out of the way when the team capability and circumstances allow. They can also help ensure that procedures are correctly followed in detail, for example related documentation maintained and correct architecture for the project is used consistently.

    Documentation and Planning

    The basic fundamentals of the project need documentation. The team concentrates more on functionality than planning but time is spent measuring the velocity and time taken on different tasks so more intelligent decisions about this direction can be made in the future. Also writing this down allows the team to look back over history and compare what they know now with what appeared to be a good idea then. Sometimes this can be helpful, however its important that any documentation that the team does produce, is kept up to date.

    What the team leader must do is help shield the team from management and too much time pressure, help ensure Agile practices are used, but not try to plan every last detail, functional or time wise (saying you are Agile because you don’t do any design or planning and don’t have team leaders isn’t Agile, its a guaranteed way of projects to fail, or at least take a very very long time to complete). Perhaps in these days its just the name – the team leader is possibly a senior developer or designer / architect, but as I have outlined in this post, I believe the requirements are subtly different.

    What do you Think?

    I would be very interested in hearing the readers views, although I’m not sure just who is reading this blog as yet. My brothers dog Boo, is about the only one I think. I’ll ask him later…

    Meanwhile, I will keep this blog updated with my experiences, because I expect I will be working on quite a few more Agile projects over the coming years.

  • The Dangers of Planning Time on Software Projects

    The Dangers of Planning Time on Software Projects

    Agile techniques favour responding to change over following a plan. I can’t remember where I read it, but I took it years ago that Agile was about staying focused on user functionality over and above trying to plan every last detail because it had been acknowledged that it isn’t possible to do that in a changing environment. Time planning is the subject of this post.

    The Dangers of Planning Time

    I have worked for many customers where the primary focus of management is the time taken by developers to create the software. As though development is some kind of production task. I believe that the only production we do is publish the project or cut the cd at the end of development. The reminder is design, not production.

    The danger of trying to plan time taken over functionality, is that faced with having to justify every minute spent, developers (especially if not architecture aware) could take short-cuts that compromise the architecture.

    This sounds crazy as I write it but it happens. If the culture is for developers to look good by getting work done quicker than others, some are more likely to take short-cuts that result in less code but poorer architecture with more dependencies. The end result is the software is quickly reduced to a complex tangled mess. The same developers can then go on to defend their actions, e.g. that they created performance improvement that obfuscated the code when they had not tested the benefits beforehand. This makes them sound good to their managers who think they are working with an expert when the reality is that the developer needs further training as they are just making a big mess.

    This can all come from focussing on time rather than functionality. In the Agile world, a complete feature can be removed from a sprint if time is short. Or a more aware developer could take a short-cut that does not compromise the core architecture. Agile techniques have more than one level of protection against this type of practice which I may blog about in future.

    Example

    An example of the dangers of planning time and trying to control it to the nth degree is as follows: Say for example, the managers say that they are “Agile”, they will allow for an hour of code review every four hours spent on coding. Sounds reasonable? (apart from this isn’t Agile and is totally arbitrary)??

    The programmers work on this basis, filling out their time if no review is required, so they don’t get accused of not reviewing the code properly, or skimping on reviews if there isn’t enough time. I don’t need to say what the end result is.

    The correct way to do this is to give the team as much time as they need for reviews, and measure the time taken. Thus as a manager you can learn the true velocity and the true situation from the times actually taken and if the software actually stands up subsequently.

    I would say that as a developer, if you can’t find a WTF in five minutes of looking, you are most probably done, so reviews should take as long as needed until you reach that point…