Omnext white paper, March 2010 How to save on software maintenance costs Abstract The cost of software maintenance is rising dramatically and it has been estimated in [1] that nowadays software maintenance accounts for more than 90% of the total cost of software, whereas it was around 50% a couple of decades ago. There are several reasons for this. We can see that software systems become hard to maintain over time, while every day more and more software is being produced. Documentation is lacking or incomplete and the people who know the software leave or retire without being replaced. Furthermore, systems tend to become increasingly complex. This makes extending a system difficult and costly. Rebuilding a system is usually not an option because the system that needs to be replaced is large, the test coverage unknown and the original and modified requirements are not well documented. 100% = Total cost of software 100% Maintenance costs 50% 1970 2009 Figure 1: Development of Software maintenance costs as percentage of total cost Given the enormous costs and efforts involved in software maintenance, every company should consider ways to make savings here, as also observed in [15]. This paper presents several known ways to save on software maintenance costs, and it also presents an integrated approach developed by Omnext. Page 1 of 12 Introduction and cost drivers Several best practices for the process optimization of software maintenance activities can be applied, such as CMMI [10], ITIL [23] and ASL/BISL [12]. All of these focus on the optimization of processes. What is at least as interesting, however, is where the real costs of software maintenance lie. Until recently [2, slide 6, 2002 and 6, 2008] relatively little was known about the software maintenance process and the factors that influence the cost of it. An investigation of recent reports about the costs of software maintenance has uncovered the following. The costs of understanding Maintenance consists of a number of different activities, as shown in figure 2. One of the key issues that needs to be dealt with is limited understanding. Limited understanding refers to how quickly a software engineer can understand where to make a change or correction in software. Some 40-60% of the maintenance effort is devoted to this task, as is noted in the Guide to the Software Engineering Body of Knowledge (SWEBOK) [3].Thus, the topic of software comprehension is of great interest to the software world. Figure 2: Analyzing is almost 50% of the total maintenance effort The costs of ‘lack of’ or ‘little’ documentation One way to remedy limited understanding is to produce documentation and other supporting descriptions such as data- or UML-models. However, how much documentation is enough? This topic was investigated during the twenty-third annual international conference on the design of communication [13, 2005], which reached the following conclusion: The surveys confirmed that source code and comments are the most important artifacts for understanding a system to be maintained. Data model and requirement description were other important artifacts. Surprisingly, and contrary to what we found in the literature, architectural models and other general views of the system were not found to be very important. How to save on software maintenance costs Page 2 of 12 The two central costs of a developer’s effort In [6] it is argued that software engineering practices can be improved if they address factors that have been shown empirically to affect a developer’s effort during software evolution. The two central finding were that: 1. The volatility of requirements had a large and consistently negative effect on effort. 2. The dispersion of changed code (distribution over different files/components) also had a large and consistently negative effect on change effort. Typically, the change requests were described by referencing a user scenario, i.e. a sequence of interactions between the user and the system, and by requesting a change to that scenario. The developers expressed that considerable time was spent understanding relevant, intermediate code when it was dispersed over many files. In other words, the qualitative data shows that when code involved in the changed user scenario was dispersed over many components that user scenario was difficult to change. Other costs involved Other cost factors in software maintenance are, among others, lack of programming standards and guidelines, solving bugs and problems, implementing changes, testing, impact analysis and integration with other systems. Any speedup in these areas would help reduce costs. The above explains what is known about the aspects that influence the costs of operational software maintenance. The next sections will be a summary of some of the practical cost savings possibilities presented in the literature. Six ways to save (Re)documentation Most existing systems are often poorly documented [5, 9]. Below we refer to several case studies where money could be saved by (re)documentation. • In one of the few controlled experiments relating to this topic, research [5] has shown that the programmers who had no documentation spent 21.5% longer understanding the code. Knowing that understanding may take around 50% of the total costs of software maintenance, we could save 12% of the total cost of software maintenance if we had documentation in place. Another particularly interesting result was that, without documentation, differences in skill levels between good and bad programmers disappeared. With documentation, good programmers did better work than poor programmers did. Without documentation, they both did equally bad work. Consequently, spending money on the best people is a waste without documentation. How to save on software maintenance costs Page 3 of 12 • Other gains of (re)documentation are reported in [18] where the break-even point occurred after 1.5 years of effort, which means that the extra investment in (re)documentation started paying off after 1.5 years of (re)documentation activity. • In [6] and also quoted in our introduction, it is said that the need to understand dispersed code is one of the two main cost factors of software maintenance, so having documentation of the dispersed code along a user scenario would significantly help. Eliminating dead code It is reported [16, 17] that up to 30% of the software in an older system can be dead code. By eliminating dead code, we reduce the code size. For an introduction to eliminating dead code and why it is important, we refer to [16]: Dead code Dead code means unnecessary, inoperative code that can be removed without affecting the program functionality. Dead code includes functions and sub-programs that are never called, properties that are never read or written, and constants and enums that are never referenced. Variables should be both read and written to. User-defined types can also be dead and a project may contain redundant API declarations. Even entire modules and classes can be completely redundant. The opposite of dead code is live, operational code. There are also several types of semi-dead code, that is, live-looking code and controls that are not actually required at run-time. Below we present several reasons for why we should care about dead code. • Dead code is a very common phenomenon. It can account for 30-40% of a project's size and increase the EXE or DLL file size by hundreds of kilobytes. The average amount of dead code is 15%. The more developers there are, the tighter the deadlines, the older and larger the system, the more dead code there is. • Dead code means excessive memory use and slower execution. It also means more code to read and maintain. It leads to higher costs, especially during maintenance. If the code is present, programmers must spend time understanding it. • Leaving dead code in a completed project means carrying around untested code and hidden bugs. Dead code may inadvertently become operative later, meaning a possible source for errors. • If you are considering the purchase of someone's source, make sure that you are not paying for hundreds of thousands of unnecessary lines. If you just took on a new project, the first thing to do should be to remove any significant dead parts. If you are finishing your newest release, remove the dead bits to ease future maintenance efforts. As reported in [19], the savings can be significant. It will cost at least $1.00 per source line per year to maintain application software. Costs for the maintenance of both dead code as well as active code are therefore estimated to be at least $1.00 per source line. How to save on software maintenance costs Page 4 of 12 Eliminating cloned code When applying new functionality, many programmers cannot resist the temptation to copy (clone) and customize existing pieces of code. The problem with this approach is that bugs are copied as well and future modifications may have to be applied to each clone, which is an error-prone procedure. Large software systems typically contain 10-25% of such cloned code. This introduces extra maintenance costs above the standard $1.00 per source line. The number of clones can be reduced by isolating the clone code in a (possibly parameterized) subprogram. It is reported in [19] that considerable savings on maintenance can be achieved by keeping the amount of cloned code to a minimum. We quote them below. Cloned code Large software systems typically contain 10-25% redundant code. This redundancy is caused by the common programming practice of replicating (or cloning) existing code and then customizing to handle new demands on an application. An IT organization consequently spends corresponding amounts of its budget redundantly maintaining this code; a bug in one code fragment is also a bug in all of its hidden clones. If redundant code were removed, the IT organization could spend the budget savings on solving new problems, enhancing the efficiency of the parent organization. The savings can be significant, because it typically costs, as stated earlier in reference to dead code elimination, at least $1.00 per source line per year to keep application software in running order. Avoiding and eliminating bugs Avoiding and eliminating bugs are on the rise. The following is noted in [20]. Bugs Bugs in software are costly and difficult to find and fix. In recent years, many tools (like Bandera, ESC/Java2, FindBugs, JLint, and PMD) and techniques have been developed for automatically finding bugs by analyzing source code or intermediate code statically (at compile time). These tools look for bug patterns which are code idioms that are often errors or potential problems. Sometimes these patterns are covered by local Standards and Guidelines or programming rules. An example is an empty catch block. In code with an empty catch block an exception is caught, but nothing is done. In most circumstances, this swallows an exception which should either be acted on or reported. Another example is an empty if statement. In an empty if statement a condition is checked but nothing is done about it. Either remove them or act upon them. In [20], Hovemeyer and Pugh relate their experiences of applying bug pattern detectors to programs. They have drawn several interesting conclusions about why you should apply them. How to save on software maintenance costs Page 5 of 12 …..“First, we have found that even well tested code written by experts contains a surprising number of obvious bugs. Second, Java (and similar languages) has many language features and APIs which are prone to misuse. Third, we found that simple automatic techniques can be effective at countering the impact of both ordinary mistakes and misunderstood language features.”…. The above experience combined with the rule of thumb often adhered to that fixing one bug costs about $1.000 makes preventing or eliminating bugs an attractive option. Focused test activities In [21], regression testing techniques are discussed and classified. The authors point out that the “retest all method” is a conventional method for regression testing, in which all the tests in the existing test suite are rerun. This “retest all” technique has proven very expensive compared with other techniques they discuss. At the heart of these techniques lies the observation that unmodified parts of programs need less test attention. So only focusing on modified parts leads to less regression test time and therefore substantial cost savings. In general: Reducing complexity leads to improved maintainability In [4] it is argued that considerable savings can be achieved by monitoring and improving the technical quality of a system before and during maintenance. This is also confirmed in [22], where the following conclusion is drawn: On the basis of software maintenance projects in a commercial application environment, it was confirmed that software maintenance costs are significantly affected by the levels of existing software complexity. The analysis on this site suggests that high levels of software complexity account for approximately 25% maintenance costs or more than 17% of total life-cycle costs. Given the extremely high cost of maintenance in commercial applications, the neglect of software complexity is potentially a serious omission. An integrated approach Thus far we have seen six possible ways to save costs and improve quality. What we are looking for is a practical integrated approach to identify these possible ways and apply them. During the past decade, Omnext has developed CARE, a concept that offers an integrated and systematic approach to cost reductions. CARE stands for Computer Aided (Re)documentation & Evaluation. At the heart of this concept lies the idea of continuous incremental improvement [much like the Kaizen [8] principle]. CARE allows you to generate savings of up to 50% on the costs of, among others, impact analysis, testing, and documenting. Furthermore, it enables you to improve the quality of your software. How to save on software maintenance costs Page 6 of 12 The concept comprises the computer-aided construction of technical and functional documentation (in the case of Omnext, this is realized through the use of its own proprietary developed technology Source2VALUE*). The concept also encompasses the automated monitoring of a software system’s quality and size. Suggestions for improvement will be made on the results that are found, suggestions for improvement will be proposed. CARE can be tailored to situation in which it is used. Figure 3: CARE (* Source2VALUE™ is a flexible technology environment that allows you to document and evaluate your systems periodically. This is done by automatically analyzing the source code, workflow definitions, menu structures, and batch-job definitions.) How to save on software maintenance costs Page 7 of 12 • Intake: A CARE environment is defined depending on the domain for which you wish to have a software documentation and software evaluation environment. The result, the CARE definition, is a clear specification of the application software sources to be analyzed, the standards, and the method of documentation and evaluation. • Monitor: Monitoring based on the CARE definition in order to analyze the application sources automatically and periodically. The result is a repository with quality and size measurements. For documentation purposes, this repository also contains all the structures and relationships of and between the application sources. The result is a data warehouse containing all the application sources and characteristics • Report: A variety of reports can be generated from the object in the repository. These may contain information about quality, size, and trend development. The technical documentation is also generated. In addition, the functional documentation that may need to be adapted is identified from the changes in the application. • Analyze: Analysis is carried out by looking at signals, quality, and productivity KPIs, and then turning the results into recommendations for improvement. The functional documentation is also analyzed from the signals. • Improve: Improvement occurs by implementing the improvement recommendations and updating the functional documentation. This updating is carried out manually or is automatically supported by tooling. If required, the CARE definition can be updated on the basis of advanced insight. We illustrate below how the ‘six ways to save’ from above can reduce maintenance costs. Source2VALUE™ Omnext’s Source2VALUE product supports (re)documentation by automatically generating technical documentation and semi-automatically generating functional documentation through the Use Case Editor (UCE). The UCE enables you to reproduce the functional working of an implemented application at a fast speed. The UCE can be deployed repeatedly, so you can guarantee that functional changes reflect the actual current source code. In summary, it provides the following functionality: • Use Case Recovery: This allows you to reproduce the functional operation of your application quickly from the source code • Business Rule Harvesting: This allows you to distill the business rules from your source code • Update signaling: This signals, on the basis of changes of source code, where you need to update your functional documentation In addition, Source2VALUE also provides insight into the quality, size, and functioning of the analyzed application. This relates to, among others, source code, workflow definitions, menu structures, and batch-job definitions. This way you are given an insight into: How to save on software maintenance costs Page 8 of 12 • • • • • • • the metrics for software quality and size the trend of the metrics for software quality and size the structure and functioning of the application the application source code (dead and cloned code) the relationships between the different application components the system workflow specification of differences in different application versions Business Benefits In the “Six ways to save” section, we presented possible savings from the literature. After applying one or more improvements, the maintenance costs will decrease, as shown in Figure 4. Note that the actual maintenance costs vary from system to system. In this case, three trend lines for the development of software maintenance costs over time are given. Software Maintenance costs CARE Time Figure 4: Effect of CARE If you are wondering what the real savings could be, Table 1 below shows a combination of possible savings, as reported in the literature. In this case, a hypothetical system of one million lines of code, with average maintenance costs of one dollar per line of code per year [19] has been chosen. Ways to save (Re)documentation Dead code Cloned code Literature source [5] [16] [19] Avoiding bugs General complexity 10 avoided bugs, General experience $500-$1000 per bug fix [22] up to 25% Table 1: Mentioned savings in literature How to save on software maintenance costs Savings in literature 10 - 25% 30 - 40% 10 - 25% Possible savings $100.000-$250.000 $300.000-$400.000 $100.000-$250.000 $5.000-$10.000 $250.000 Page 9 of 12 Note that in reality the overall possible savings will most likely be lower. However, if you suppose that even the saving possibilities that we have mentioned result in an overall scenario where savings are 30% ($300.000), then you would achieve a substantial benefit in terms of costs, not to mention the positive overall effects on quality and productivity for all maintenance efforts. Summary We have presented six ways from the literature to save software maintenance costs. These are often based on empirical studies. We then described the Omnext CARE concept, a practical solution for establishing continuous incremental improvement and thus lowering software maintenance costs. Furthermore, we described the unique state-of-the-art capabilities of Omnext’s Source2VALUE technology. Finally, using an integrated approach, we presented the financial benefits for your business, which, as we have seen, can be substantial. Call to action After reading this paper one thing should be clear. Saving on maintenance costs will never be achieved with one major action. It calls for an integrated approach, preferably institutionalized in the company’s own software maintenance area. Practical information on software system(s) combined with the right supportive technology and the right approach (where relevant aspects get their attention) is needed. Only then can software maintenance costs be lowered and kept low in a sustainable way. Authors Ir. Floris P. Engelbertink, Omnext, e-mail: floris.engelbertink@omnext.net Dr. Harald H. Vogt, Omnext, e-mail: harald.vogt@omnext.net References and links 1. Software Maintenance Costs, Ó Jussi Koskinen, Information Technology Research Institute, ELTIS-project, University of Jyväskylä, Finland, http://users.jyu.fi/~koskinen/smcosts.htm 2. Estimating Software Maintenance, Seminar 2002, Arun Mukhija, Institut fur Informatik der Universität Zurich, http://www.ifi.uzh.ch/rerg/fileadmin/downloads/teaching/seminars/seminar_ws0203/S lides_9.pdf 3. Guide to the Software Engineering Body of Knowledge (SWEBOK), http://www.computer.org/portal/web/swebok/html/ch6#Ref1.1 4. Softwaremonitoring, Lagere TCO door kwaliteitscontroles, SIG, http://www.softwareimprovers.com/blobs/Publicaties/Publicaties%202009/20091104%20Lagere%20TCO%20 door%20kwaliteitscontrole.pdf 5. Report from an experiment : impact of documentation on maintenance in Journal of empirical software engineering, Tryggeseth, Eirik, Kluwer Academic 2: 2, 201-207 1997 ISSN 1382-3256, http://www.realisation.com.au/site1/Articles/Reducing%20software%20maintenance% 20costs.htm 6. An investigation of change effort in two evolving software systems, Technical report 11/2008, Simula Research Laboratory, related paper appeared as “Understanding cost How to save on software maintenance costs Page 10 of 12 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. drivers of software evolution: a quantitative and qualitative investigation of change effort in two evolving software systems” in Journal Empirical Software Engineering, http://simula.no/research/se/publications/Simula.SE.317/simula_pdf_file Software Evolution, Universität Bern, Lecture, Tudor Gîrba, http://scg.unibe.ch/download/evo/01-introduction.key.pdf Kaizen, The Japanese Strategy of Continuous Improvement, http://www.1000ventures.com/business_guide/mgmt_kaizen_main.html A Survey on the Software Maintenance Process , Proceedings of the International Conference on Software Maintenance, Maria Joao Sousa, Page: 265, Year of Publication: 1998, http://portal.acm.org/citation.cfm?id=853300 CMMI for Development, Version 1.2 , Technical Report, CMU/SEI-2006-TR-008, Software Engineering Institute, http://www.sei.cmu.edu/library/abstracts/reports/06tr008.cfm ASL BiSL Foundation, Website, http://www.aslbislfoundation.org/component/option,com_docman/task,cat_view/gid,1 0/Itemid,24/lang,nl/ CMMI: Overstappen, waarom en hoe?, Henk Westerink, Rini van Solingen en Eric van der Vliet , LogicaCMG, http://www.dse.nl/~thelosen/artikelen/Final%20CMMI%20Overstappen%20waarom%2 0en%20hoe.pdf A study of the documentation essential to software maintenance, ACM Special Interest Group for Design of Communication, Proceedings of the 23rd annual international conference on Design of communication: documenting & designing for pervasive information, Sergio Cozzetti B. de Souza , Nicolas Anquetil, Káthia M. de Oliveira Universidade Católica de Brasília, Brasilia, Brazil http://portal.acm.org/citation.cfm?id=1085331&dl=GUIDE&coll=GUIDE&CFID=72997626 &CFTOKEN=16700333 The Impact of UML Documentation on Software Maintenance: An Experimental Evaluation, Simula Technical Report 2005-14 and Carleton Technical Report SCE-05-11, Version 2, http://134.117.61.33/pubs/tech_report/TR_SCE-05-11-ver2.pdf The economics of software maintenance in the twenty first century, Capers Jones, Chief Scientist Emeritus, Software Productivity Research, Inc. http://www.compaid.com/caiinternet/ezine/capersjones-maintenance.pdf Aivosto — Programming Tools for Software Developers, Website, http://www.aivosto.com/project/deadcode.html Architectural Modifications to Deployed Software, A.S. Klusener and R. Lammel and C. Verhoef, SIG and Free University of Amsterdam, http://homepages.cwi.nl/~ralf/am/paper.pdf. A Case Study on the Long-Term Effects of Software Redocumentation , ICSM, Proceedings of the 20th IEEE International Conference on Software Maintenance, Alexander J. Rostkowycz , Vaclav Rajlich, Andrian Marcus, Wayne State University, Pages: 92 – 101, Year of Publication: 2004 http://portal.acm.org/citation.cfm?id=1021425 Semantic Designs, Website, http://www.semdesigns.com/Products/Clone/ Finding Bugs is Easy, David Hovemeyer and William Pugh, Dept. of Computer Science, University of Maryland, http://www.cs.nyu.edu/~lharris/papers/findbugsPaper.pdf Understanding regression testing techniques, Gaurav Duggal, Mrs. Bharti Suri, Guru Gobind Singh Indraprastha University, Delhi, India. http://www.rimtengg.com/coit2008/proceedings/SW15.pdf Software Complexity and Maintenance Costs, Communications of the ACM, Banker et al, November 1993, How to save on software maintenance costs Page 11 of 12 23. 24. 25. 26. http://www.pitt.edu/~ckemerer/CK%20research%20papers/SwComplexityAndMaintena nceCost_BankerDatar93.pdf Official ITIL Website, http://www.itil-officialsite.com Het legacy probleem aangepakt, H. Vogt & M. Maat, 1998, www.serc.nl Een methodische aanpak voor Legacy, H. Vogt, Informatie, September 2004, http://www.omnext.net/downloads/een_methodische_aanpak_voor_legacy.pdf Quantitative IT Portfolio Management, C. Verhoef, 2002, Science of Computer Programming (45), http://www.few.vu.nl/~x/ipm/ipm.html How to save on software maintenance costs Page 12 of 12