Figure 1a. Growth in spacecraft onboard data processing (PROM +RAM) Courtesy of T. Vardenaga, ESA
The increasing general importance and scope of software applications is evident in almost every facet of modern life. There have been dramatic advances in computer technology and processing capability, with even the PCs to be found in many households having a processing capacity and range of sophisticated software which surpasses much of what was to be found in commerce and industry just a few years ago. It is therefore hardly surprising that in a leading-edge, high- technology activity like the space sector there has been a similarly marked increase in software applications and associated expenditure. This trend seems set to continue, in all probability with increasing momentum.
Lederer & Prasad, Nine Management Guidelines for Better Cost Estimating, ACM, Vol. 33, No. 2.
Much of the commercial software available for general business use has been developed at enormous cost, but this is justified by the promise of widespread application and sales. There is, however, much software which is developed for very specific applications with special needs and characteristics. This is the case with most software development for the space sector, whether it be for on-board, in-space applications, which certainly have some unique characteristics, or whether it be for ground-segment applications for which the requirements may be different but the applications somewhat more extensive. There is also some specific software development for administrative purposes, but whilst the cost of such developments may be substantial, it is relatively insignificant compared with that of software development for the technical applications.
The proportion of the ESA budget spent on software- development contracts for its programmes has doubled in the last decade. The reasons for this growth are to be found in the predominance of the role played by the amount of processed information generated by each ESA spacecraft, and the ever- growing autonomy of these spacecraft (Fig. 1).
Figure 1a. Growth in spacecraft onboard data processing (PROM
+RAM) Courtesy of T. Vardenaga, ESA
Figure 1b. Growth in ground-segment software Courtesy of J. Bui
et al., US Institute for Defense
Today, expenditure on ground-segment software can represent some 40% of the related budget, whilst for the space segment expenditure on on-board software and ground test systems can represent in excess of 10% of that element's budget.
It is notoriously difficult to accurately predict software development costs, this being a problem both for those companies who actually carry out the development, and for customers such as the Agency who want to be sure that estimates of the effort required are credible and that prices are reasonable. A recent survey* has demonstrated a crucial challenge in this respect in that it was found that two-thirds of all major software projects have substantially overrun both their schedule and cost estimates.
Although more than 20 software cost-estimation models were already in existence as early as 1981, the survey revealed that in practice many software-development cost estimators still tend to rely rather more on their personal memory of the cost of similar projects than on use of other estimating processes. This approach can certainly not be relied upon, and the study concluded that this intuitive approach correlated positively with the percentage of large projects overrunning their estimates. Only the use of data for similar past projects based on documented facts, the use of simple arithmetic formulae, and the use of established standards, were found to lead to greater accuracy in software cost/effort estimates.
From the beginning, the estimation of software development effort has never been easy. Firstly, the abstract nature of software makes it a difficult product to characterise, and design changes are often introduced into software before, during and after its pure code-production phase. Secondly, there are often too few projects in process within a particular organisation to provide a good basis for estimating the cost of new developments. Lastly, software production has evolved in recent years from being something of a black art, into an engineering discipline in which there is a very rapid evolution of techniques. The momentum of change in this field therefore makes the process of basing new estimates on anything other than fairly recent developments very suspect. Consequently, the collection of software metrics is now viewed as the best basis for increasing software production efficiency and providing a rational means of measuring and estimating development costs. Such metrics allow the application of statistical-analysis techniques to generate the mathematical expressions needed as a basis for 'parametric cost models'.**
** Sets of mathematical equations describing the relationships between cost, schedule and measurable attributes of a system.
This article describes how ESA has organised a continuous gathering of historical data on software-development programmes from diverse sources, including sources outside the space sector. The latter are of interest because they provide the key to calculating the effect of specific characteristics of space software, such as the very high reliability required, the need to work first time, the need to be able to effect in-flight repairs and maintenance, and the specialised microprocessors on which it runs. This information has been analysed and the results distributed to the participating organisations, all of whom are interested in gaining a better understanding of the process and are seeking to be able to predict the effort required for their future projects more accurately.
In 1988, the Cost Engineering Section of the ESA Cost Analysis Division, faced with the evaluation of proposals for large space programmes and the ever-increasing software-development element, began com-piling a software metrics database focusing on effort and productivity measures. Industry was encouraged to participate in this initiative on the basis that by contributing 'productivity oriented' data they would receive in return the whole set of project case studies and a basic analysis of the data. Subsequently, the database was expanded to include military and industrial applications.
At the end of 1993, it was felt that this exercise should be intensified, which would necessitate additional administrative and management support. It was decided that INSEAD, as a leading international business school, would be an ideal partner with which to collaborate in this endeavour. Firstly, INSEAD would be seen by potential contributors as a neutral body, and not as a potential business competitor. Secondly, INSEAD was already conducting a large annual survey of European manufacturing industries, the survey itself involving techniques similar to the approach being followed for the software metrics exercise. Thirdly, researchers at INSEAD are involved in a number of other research projects designed to respond to the business and managerial challenges associated with achieving excellence in software, and there is thus some synergy between these activities and the ESA initiative. For its part, INSEAD was happy to participate in this task in order to gain a better understanding of the software-development process, with the aim of improving management practices in this area and determining performance benchmarks.
A comparable data-gathering exercise was initiated in the USA by the Space System Cost Analysis Group (SSCAG), whose members include US government agencies such as NASA and the Department of Defense (DOD), the largest space industry contractors (both American and European), as well as ESA. Subsequently, the databases on both sides of the Atlantic have been compared.
By mid-1995, the ESA Software Development Database contained information about 108 space, military and industrial projects from 37 companies in 8 European countries. The information is compiled by INSEAD, which contacts each supplier of data on a regular basis to determine if suitable projects are nearing completion. When a completed questionnaire is received, each data supplier is telephoned to ensure the validity and comparability of questionnaire responses. In return for supplying their data, each participating organisation or company receives the periodic data- analysis reports and also the complete dataset. Elements related to the source of each project are not divulged, but each data supplier is given the means to identify their own data, to allow benchmarking against the rest of the dataset.
The 108 projects represent 5.51 million lines of source code (range 2000-413 000; average 51 010), 22 development languages or combinations of languages, and 30 125 man months of effort (range 7.8-4361; average 284). Further details about the database are given in Figure 2 and Table 1.
Figure 2a. Percentage of projects by main application environment
Figure 2b. Percentage of projects by country
Figure 2c. Percentage of projects by language
Figure 2d. Percentage of projects per company (ten largest data
suppliers)
Table 1. Variables in the ESA dataset
LANG Application Programming Language
ADA Ada
PAS Pascal
FOR Fortran
LTR LTR
C C
TAL TAL
COR Coral
AS Assembler
ENV Environment (Space, Military, Industry)
COMPANY Company where project was developed
COUNTRY Country where project was developed
CATEGORY ESA Classification
OB: On Board
MSG: Message Switching
RT: Real Time
GSE: Ground Support Equipment
SIM: Simulators
GRD: Ground Control
TL: Tool
OTH: Other
TEAM Maximum size of implementation team
DUR Duration of project in months
KLOC Kilo Lines of Code
YEAR Start Year of Project
RELY Required Software Reliability
TIME Execution Time Constraint
STOR Main Storage Constraint
VIRT Virtual Machine Volatility
LEXP Programming Language Experience
MODP Use of Modern Programming Practices
TOOL Use of Software Tools
In the ESA database, 'KLOC', 'Effort' and 'Productivity' are defined as follows:
Some general results of the productivity and effort estimation statistical analysis undertaken by the INSEAD research team are presented below. This study, whose details are available exclusively to the companies participating in the survey, is thought to be unique in that no other published research analysing such a large number of European non-MIS (Management Information System) applications can presently be identified.
Over the past years, many effort estimation models have been developed. However, because of differences in data collected, project types and environmental factors between software development sites, these models are generally only valid within the organisation in which they were developed, or after conducting extensive calibration exercises. One major obstacle to the transportability of these models appears to be a lack of understanding of the factors explaining the differences in productivity between projects. As there are literally hundreds of parameters that can affect software development effort, and as only a few of them may affect the productivity within a particular environment, it is important to analyse the accuracy of these effort estimation models based on the prior determination of the factors found to explain the productivity of projects in a given database. Furthermore, it is also important to determine whether effort estimation models based on the significant productivity factors of a multi-company data-base can be successfully used to make effort estimates within a specific company.
The first step in the analysis was to identify the factors that explain the productivity differences in the data. The factors considered were: company, country, language, category, environment, start year, team size, project duration and system size, as well as the following seven COCOMO* factors: required software reliability, execution time constraint, main storage constraint, virtual machine volatility, programming language experience, use of modern programming practices, and use of software tools. Simple empirical effort estimation models were then developed based on these productivity factors.
* The Constructive COst Model is a major software engineering economics model developed by TRW in the United States.
As organisational differences account for most of the productivity variation between projects in the ESA dataset, it was decided to develop general ESA effort estimation models with the data from one company removed, and then to test the general ESA models on the removed company. Simple company-specific effort estimation models based only on the factors found to affect the productivity of this particular company were also developed. The results of the best company-specific models were then compared with the models developed using the ESA dataset (Fig. 3).
Figure 3. Effort estimation model development and testing on
databases
As noted above, organisational differences were found to account for most of the productivity variations between projects in the dataset. This highlights the need for companies to establish their own software metrics databases, in addition to benchmarking their data against that of other companies. For benchmarking purposes, the best classification of the productivity of European space, military and industrial projects was determined to be by application, category and programming language.
It can also be concluded that accurate effort estimation is possible with just a very small number of productivity factors. Simple effort estimation models based on the size, in kilo-lines of delivered source code (KLOC), and the significant productivity factors of a given database are reasonably accurate when applied to projects from that database. However, effort predictions made on an individual company's software projects using a general model based on a multi-company database were shown to be less accurate than predictions made using a company-specific model. This underlines the importance of conducting further research into the appropriate set of productivity factors to be tracked in multi-company databases.
The benefits of this research to ESA are strongly linked to the benefits obtained by the contributing companies. The Cost Engineering Section needs this software development database to fine-tune the parametric models in its cost-estimation tool and to support cost estimation by analogy when preparing cost estimates for new ESA projects. As basing new estimates on anything but comparatively recent developments is somewhat precarious, a continuous input of new projects is needed to keep the database up-to-date.
Clearly, companies are more likely to supply this data if there is an advantage to be gained by doing so. This research has shown that organisations can indeed benefit from contributing to multi-company software development databases, especially those that cover their particular application areas, through an increased understanding of the factors that influence the productivity of similar projects across companies and through the opportunity to benchmark their software development productivity. The analysis of this confidential data by an independant third party can lead to a greater knowledge for ESA and all contributing companies than would otherwise be the case.
As previously indicated, it is desirable to maintain the present activity so as to have a better understanding of the process and to keep up to date in a rapidly changing domain. The participation of additional companies is sought in order to provide a wider basis for the analysis. Within ESA itself, with so many groups directly or indirectly involved in software-related activities, it is also most important to communicate and exchange ideas and information so as to optimise effort in this area.
A further aspect to be developed is the life-cycle approach, with software-maintenance costs being a very significant part of the overall cost of software development and application. Contacts with professional bodies such as the SSCAG and the ISPA (the US-based International Society of Parametric Analysts, which has many European members from the space sector) are important as a source of ideas and for the exchanging of both information and experience, because with limited resources available for such initiatives it is important to focus on the novel aspects and certainly to avoid 're-inventing the wheel'.
ESA Bulletin Nr. 87.