Figure 1. On-board software and operations
The Cluster on-board software will handle primary spacecraft control, as well as the science data acquisition and formatting. To ensure safe operations and service continuity, the On-Board Software Maintenance (OBSM) concept that will be employed allows in-flight modification of the running software, through an incremental model based on patches. New programs are compared to the current software on board and the differences are converted into memory-load commands. Four independent versions of the on-board software will be maintained - one per spacecraft - with multi- and inter-spacecraft handling for the common functionalities.
The necessary OBSM tools have been set up at ESOC to support the production of the software updates and their uplinking to the four spacecraft, and to keep track of the prevailing configurations. In addition, ESOC has developed several application programs that will facilitate routine Cluster operations.
Why on-board software maintenance?
With recent generations of spacecraft, complex onboard software has emerged as a key component between the ground control facilities and the spacecraft platform and payload being operated, by offering a high level of spacecraft autonomy and flexibility. This software can be modified or even redefined during the mission, whereas most other on-board subsystems are confined within the limits of pre-defined configurations.
The need for On-Board Software Maintenance can arise at any time, from immediately after launch if the spacecraft's post-launch status calls for operational adaptations, until the end of the mission when ageing of the in-orbit hardware leads to a greater probability of failures. Even if these OBSM services are not used regularly, an adequate infrastructure must be available on the ground throughout the mission to correct, add to or re-design the on-board software whenever the need arises.
New programs must be uplinked to the spacecraft during the mission in order to adapt the OBDH's behaviour to the prevailing status of the spacecraft hardware and any operational difficulties that are encountered, and sometimes to satisfy additional requirements for the benefit of the user community or for operational safety reasons.
The operational environment
The software on-board the spacecraft is used by the spacecraft operator to run the mission according to specific rules (Fig. 1). These rules are prescribed flight and control procedures prepared by ESOC for each space mission, incorporated in the Flight Operations Plan, and adapted during flight as necessary. OBSM calls for specific flight control procedures for adding or modifying on-board functionality, or for restoring a known safe status. As such OBSM activities can change the OBDH subsystem's overall behaviour, configuration rules are introduced to control the successive software versions.
Figure 1. On-board software and operations
The OBSM tools have to support both the production of new software and its uplinking to the OBDH subsystem, whilst also ensuring safe integration of the new elements with the flight software originally delivered by the spacecraft manufacturer and keeping track of what is actually loaded on-board at any given time. An OBSM workbench integrated into the operations environment has to include a compiler and software development environment (SDE), a software validation facility (SVF), a generator for patch telecommands, and a mission database of "memory images". The latter are files representing, on the ground, the contents of the spacecraft's on-board computer memory (programs and data) at any time during the mission.
ESOC's role
Industry cannot be expected to provide all of the expertise or to maintain at their premises all of the tools that might be needed at very short notice at any time during the routine phase of every space mission, some of which can last for ten years or more. Keeping on-board software experts from industry on standby at ESOC throughout such missions would be just too expensive, not to mention the difficulties that would be encountered when missions are extended or terminated at short notice. Another alternative of buying off-the-shelf development tools from the OBDH manufacturer would neither guarantee the timely availability of adequate OBSM facilities nor the availability of the requisite expertise. Such development tools are not necessarily suitable for software maintenance and tend to be geared to hardware and software configurations far removed from the ESOC norm in terms of standards. Moreover, the spacecraft operator must be totally familiar with the on-board software and its current configuration, making close integration of the software tools within the ESOC environment mandatory. Hence, it is desirable to have an autonomous OBSM infrastructure within the Control Centre itself, where the operations rules are defined and routinely applied. Within ESOC, the Mission Operations Department is responsible for OBSM, and the OBSM engineer forms part of the Flight Control team.
The choices for Cluster
This scientific mission involves four classical spin-stabilised spacecraft, each with its own ESA "standard" OBDH subsystem, running 72 kB of flight software. The scope of the OBSM is limited to maintenance of the OBDH software, the payload computers being the responsibility of the scientific investigators.
Prior to launch, the spacecraft manufacturer Dornier (D) is responsible for any adaptations of the on-board software. Any updates needed after the final delivery and prior to launch will be provided in the form of software patches, which ESA can decide to apply or not after test and trade-off with operational workarounds.
ESOC will be responsible for the Cluster OBSM from the moment of launch and a support contract for the Launch, Commissioning and Mission Phase (Phase-E/F) has been established between ESA and the Cluster OBDH supplier Laben (I). During the launch and commissioning of the spacecraft platform, Laben's OBDH experts will be available at ESOC. Thereafter, ESOC will be in the front line for the duration of the mission, with the contractor providing second-line OBSM support based on its reference knowledge of the multi-layered Cluster OBDH software and hardware and its development (Fig. 2). All implementation, compilation, linking and configuration-control tasks will be under ESOC's responsibility.
Figure 2. On-board layering of Cluster software
Three major features of the OBSM concept for the Cluster mission are: the ability to modify programs whilst they are running; the ability to distribute common software elements to several spacecraft; and the routine maintenance of operations data.
Live program patching
The OBSM concept originally defined by ESOC assumes a global and linear process. New programs and modifications to programs are written in ADA in a Software Development Environment (SDE), compiled and linked to the current flight code. The resulting complete new executable image is then delivered to the Cluster Dedicated Control System (CDCS) where a history of the on-board memory contents is stored. The new contents are then compared with the current status of the memory and the differences, i.e. the software upgrade, are systematically converted into memory patch telecommands ready for uplinking.
The Cluster OBDH software has to run continuously to ensure spacecraft survival, but the available on-board resources do not allow the running of two software versions - one active and one being loaded or modified - simultaneously. The existing software therefore has to be upgraded whilst it is running, which excludes recompilation and linking of the whole flight source code at ADA level, since the resulting executable image might be completely different and then impossible to load.
The global OBSM concept has therefore been transformed into an incremental model based on patches, which requires a deeper knowledge of the flight software image delivered by the OBDH manufacturer. This reference image is always reloaded from PROM if a reboot occurs, and it will remain the basic framework to which all new software must be attached throughout the Cluster mission (Fig. 3).
Figure 3. Main memory areas
As new instructions added to or replacing old code will become active immediately they arrive on-board, the patching procedure must take into account the detailed logic of the program being modified in order to ensure a clean transition. In addition, independent program units can be registered as application programs in the built-in task scheduler, and activated later.
Multi-spacecraft facilities
Certain code upgrades, for instance new functions defined by ESOC, will be common to the Cluster fleet and will need identical implementation. For sound configuration control, such software items to be loaded into all four spacecraft must come from a single source. Other potential software upgrades like those compensating for hardware failures on individual spacecraft will, by definition, be spacecraft-dependent and will therefore have to be created and maintained in an environment protected from interferences with the OBSM environments of the other spacecraft.
The principle of four independent on-ground OBSM environments handling four independent computers in flight has been amended to allow inter-spacecraft copying of the contents of memory patch telecommands. It will therefore be possible to distribute a program produced in the development environment to one, two, three or all four spacecraft. However, dumped memory images will be unique to each spacecraft and cannot subsequently be exchanged.
OBSM data-handling support
OBDH reconfiguration tasks like the reprogramming of telemetry acquisition or recording tables can easily be carried out with the OBSM patching tools. Using the multi-spacecraft facilities, such tables can be reprogrammed identically on all four spacecraft, as long as no divergence in on-board hardware imposes different configurations on different spacecraft. Also, when performing OBDH check-ups, OBSM comparison tools will facilitate the automation of large data verifications between a memory dump and the expected pattern. Data maintenance will be even easier than program maintenance because the values modified on-board will generally be passive or can be de-activated during the time of the change. This facility will provide an opportunity to exercise the OBSM procedures and tools, so that when live program patching is eventually required basic confidence in the techniques will have already been established and all efforts can be focused on defining and validating the necessary software modifications.
The primary OBSM tools for the Cluster mission are shown in the schematic of Figure 4. The two Software Validation Facilities - labelled SdeVF and SimVF - correspond to the two types of real-time requirements during the validation of new software, as explained below.
Figure 4. Primary OBSM tools
Software Development and Validation Facility (SdeVF)
An innovative Software Development and Validation Facility has been provided to ESOC by ESTEC as the result of seeking a cost-effective approach to serve both the Soho and Cluster missions. This facility offers a unique trade-off between low-level and high-level validation tools, combining two major validation strategies: software debugging on target hardware, and telecommand/telemetry consistency checking (Fig. 5).
Figure 5. A single tool with which to develop and validate
At one level, the SdeVF serves as a development environment, including a 1750A Assembler compiler, for writing application programs and patches in Assembler. New programs and corrections can be immediately downloaded to the emulated target hardware, a 1750A microprocessor identical to that embedded in the Cluster OBDH, on which the rest of the flight software is already running. As from the coding step, therefore, this analytical tool helps in taking care of integration constraints, like available memory space, timing, performance and existing code.
At the next level, the SdeVF serves as a validation facility, with which the primary interaction of the on-board software with its spacecraft environment can be checked. Cluster-specific functions simulate the OBDH hardware items like the telecommand decoder and the telemetry frame generator. The spacecraft outside the OBDH is simulated by a logic derived from the database of the Electrical Ground Support Equipment (EGSE): the telemetry reflects the status of devices switched on or off by telecommands. The evolution of analogue parameters like temperatures can be simulated stepwise, devices can be failed to a blocked status, etc. The simulated spacecraft is operated through the EGSE telecommand and telemetry interfaces in an interactive mode.
With its test sequencer, the SdeVF provides a major step towards the automation of software validation in that it is possible to program interactively every verification in a test scenario. The whole set can then be replayed by the test sequencer in a matter of hours to provide a formal go/no-go verification for the acceptance of a new on-board software release.
Interactive OBSM
Control tools used at ESOC to handle memory images and patch commands for the Agency's two ERS remote-sensing spacecraft have been adapted for Cluster, resulting in the Cluster OBsm Interactive (COBI) facility noted in Figure 4. COBI supports the on-ground representation and maintenance of the software images on-board the four Cluster spacecraft (Fig. 6) in four distinct environments - one per spacecraft - with inter-spacecraft facilities. As most configuration control is performed on memory images, the on-board configuration of each of the four spacecraft must always be known by the ground so that it could be reconstructed at any time if needed. The memory images are obtained from memory dumps, the interpretation of which is facilitated by symbol tables.
Figure 6. The image modification loop
Telecommands are constructed within COBI to modify the on-board software. Patch command sequences are obtained by comparing new memory sections or programs defined and validated on the SdeVF with actual on-board memory contents. The differences are converted into memory load commands, ready for uplinking. Only small portions of the memory (e.g. 1%) are affected each time, as all modifications respect the structure and address mapping of the reference flight code.
Simulator-based Validation Facility (SimVF)
This is a mode of the Cluster simulator in which the OBDH software model is replaced by a hardware emulation, as in the SdeVF, with a 1750A microprocessor identical to that in the Cluster OBDH running the flight software. The SimVF combines the real - and modifiable - on-board software with a full spacecraft simulation. It is operated from the Control Centre through the operational ground environment (Fig. 7), via a simulated ground station.
Figure 7. Operational validation
Operational validation of new on-board software can thus be reliably carried out using the SimVF. The behaviour of the OBDH, including software as OBSM intends to modify it, is testable at system level within the modelled spacecraft. The robustness of the modified system can be assessed by executing operational procedures that involve the upgraded functionality.
The SimVF also supports OBSM procedure validation. As the installation of new software modifications has to take place in a continuously running software environment that ensures spacecraft survival and cannot be interrupted, such rehearsals are mandatory!
On-board software maintenance activities of the type described here have already been successfully carried out during the pre-launch phase of the original Cluster project in 1996. ESOC was able to successfully apply off-the-shelf patches provided by the spacecraft manufacturer and to add application programs to improve operational capabilities beyond those included in the original design. ESOC is therefore certainly ready to fulfil its role in providing operational On-Board Software Maintenance for the Cluster fleet of four spacecraft in simultaneous operation.
With the current trend towards standardisation of on-board computer systems, the efforts described here are undoubtedly a good investment for the future. The new software maintenance and validation techniques and procedures that have been devised for Cluster are already being applied for ESA's Huygens and XMM missions, providing further confidence in ESOC's abilities to meet the needs of ESA's ever more demanding space missions for many years to come.
ESA Bulletin Nr. 91