LinuxAsmTools
LinuxAsmTools - Is Codding hard?
For 50 years we have worked at improving the software development process and little has changed. Why is software development so hard?
This is the question posed by the book "Dreaming in Code"(published in 2007). The book states over 66% of all projects are either canaled or miss their targets by a wide margin. At the same time optimistic programmers are touting new techniques,languages, and tools that will solve development problems.Why do these new tools and techniques have little impact?
The book "Dreaming in Code" did not have any answers. They ended by saying : "coding is hard". I, on the other hand, am one of those optimistic programmers who thinks the answer is obvious. The rest of this essay presents my case.
First, my credentials. I've worked on about a 100 projects since 1967. This has included large projects, small projects, many failures, and a lot of successes. The fact that the success have some things in common and the failures also share traits gives us the clues to begin our quest. We can assume if some projects are successful then there are answers to be found.
Postmortems of failed projects have one thing in common. Lots of problems and over optimism. The problems range from budget constraints, design errors, poor skills, shifting requirements, and just about anything that could possibly go wrong. The conclusion is obvious; software projects can be complex. Thus, the answer must involve reduction and management of complexity. Seems obvious, but tell that to all the language and tool developers who felt they were solving the problem. Also, tell the methodology folks and those who pushed good programming practices.
Have we focused on the wrong problems for the last 50 years? I think we have. The answer is right in front of us and it is being ignored.
We know that the coding process is sometimes a small part of big projects. Successful projects almost universally recognize this fact. This gives us another good clue.
We also have lots of facts showing great diversity in programmer skill levels, and adding more programmers often kills a project. We know about open development projects and ego-less programming. We also know about modular programming and splitting a project into definable tasks.
Even with all this knowledge, our problems have persisted. So, what is the answer?
Where we failed is in process. The old idea of breaking a project into phases (designi, code, test, integration) is part of our problem. If we look at the problems we see design problems and integration problems throughout the life of some projects. In some cases prototype code is done before design. We are sometimes forced to do bottom-up design and sometimes we use top-down design.
What we need is a continuous and ongoing design effort that eventually verifies and tests itself against the coding effort. This is the major component of our new mind set.
The design team are the only group that can access the impact of changing requirements, design changes, problems, testing, and performance issues. This is an area that has not been given adequate attention on most projects.
For small projects by one individual, the same conclusion applies. Any project or module that can be handled by one coder,requires they wear two hats. Those hats are system engineer(design) and coder. Both hats must be worn concurrently.
OK, all this has been said before and used on projects, what's different?
What if we actually did system engineering and veiwed it as a major skill set. That would be different. We could also stop trying to simplify the job of system engineer and simplify the project instead. Obviously we have done a poor job of managing complex software projects. That can only mean we need better management skills and possibly a new model for software projects. I think both are needed.
Once again the answer is mostly process. Our design (system engineering) effort needs to build a project model and review it continuously. For a single person project, this may a simple list of tasks and measurements. For big projects it might be simulations and test stubs. It would include schedules, and anything used to evaluate progress.
The size of the design effort would be based upon the number of unknowns. The unknowns include things like hiring programmers, new tools, and possibly changes in the weather.
Small projects may spend less than 20% of the effort on design. As the project size grows, so does the design (system engineering). A large project may be over 80% system engineering.
Another big change in mind-set would be the merging of system engineering, integration, and evaluation. Often these are separate efforts. What folly, asking non designers to integrate and evaluate a design is usually a disaster. At best it is a costly and slow process. Many projects ask the programmers to do evaluation once the coding is done. This approach sometimes works, but would be better as a parallel process handled by the people who designed the project.
There is one hole in our conclusions so far. A poor design team can add complexity and unknowns to a project and make it fail.The answer is to develop standards for system-engineers similar to what can be found in many areas of engineering. We need a job classification called "software system engineer" and it needs to have basic standards.
In summary, we need to stop asking coders to do system design. A few have the skills for system design, but most do not. We also need to develop the process and knowledge base for system designers. Some questions we may want to give our prospective designers are:
When does a project need to use a design language?
How do determine the benefit of developing a prototype?
How do we measure project complexity?
How can we develop schedules if dates can not be determined?
How do we split projects into pieces and manage each task?
What are the project goals and how will they be acheived?
These are very common questions asked by projects and essential to success. It is also the area where we are currently failing.
