European Space Agency

Sharing Mission Data via the Internet

M. Merri

Mission Control System Division, Directorate of Technical and Operational Support, ESOC, Darmstadt, Germany

On reception at the Control Centre, spacecraft data is routed, together with the relevant auxiliary data, to the Data Distribution System (DDS) where it is available in near-real-time. End users (for instance, Principal Investigators or other members of the scientific community) may be allowed access to this data on-line via the Internet as well as off-line via appropriate Mass Delivery Media (MDM). The users may also be allowed to select data according to predefined mission-dependent criteria (e.g. experiment, experiment mode, time range, data quantity) and retrieve it to their local computers for further analysis. To provide a standardised format for the data delivery, on-line and off-line data may be encapsulated in a Standard Formatted Data Unit (SFDU) as per the recommendations of the Consultative Committee for Space Data Systems (CCSDS).

In support of this data distribution approach, ESA has developed a generic Control Authority Office (CAO) system* to handle the on-line registration and dissemination of data descriptions relative to the mission data that is available to external users.

* The current release of the ESA CAO system is available at http://cao.esoc.esa.de/cao-bin/cao_home to all users with World Wide Web and socket interfaces.

Introduction

Satellite missions are always performed to achieve well-defined goals. However, these goals would be incomplete if the data produced by a mission were not delivered to end users. In the past, data was often delivered on media that required postal shipment for final delivery. Recent technological developments in networking and a larger interest for networked applications are making the Internet a very attractive alternative in delivering mission data. The system used to distribute mission data to the user community is often referred to as the 'Data Distribution System (DDS)'.

Distinct advantages of using Internet technology include the speed and efficiency with which the data can be delivered, and the fact that more sophisticated data selection can be exercised by end users. This means that a user interested in a limited part of the data can electronically request and receive just that particular data set.

The Consultative Committee for Space Data Systems (CCSDS), a standardisation body that represents most space agencies, has produced recommendations for the Control Authority Office (CAO). The goal of the CAO is to make descriptions of mission data available to users in an organised and simple manner. The CCSDS concept foresees that each CCSDS-participating space agency has one or more CAOs, each managing and maintaining mission-specific data descriptions. ESOC took the lead in the context of CCSDS Panel 2 in developing a CAO system that is compliant with the CCSDS recommendations and with which the user community can interact via the Internet.

In this context, it is likely that future missions will make more and more use of the Internet to distribute mission data and to deliver the necessary services to guarantee long-term (actually indefinite) data documentation. This does not imply that the Internet will be the only mission- data transfer medium of the future. In fact, depending on the quantity of mission data that needs to be transferred, the size of the user community that wishes to access this data and how quickly the data is required at the end-user site, other media might be preferable. These media, which require physical shipment to the end user (for instance, postal mail), are referred to in this article as 'Mass Delivery Media (MDM)' and currently include compact disks (CD-ROM), digital audio tapes (DAT) and magnetic disks.

Nevertheless, even if MDM were selected by a mission as its primary data transfer medium, Internet access to a limited portion of the data might still provide useful services, for instance in allowing users to quickly inspect the quality of the data prior to ordering it on MDM. This would avoid unnecessary MDM production costs. The user could also place the MDM order via the Internet and even 'compose' a customised MDM carrying specially selected mission data.

The term 'Internet' does not exclude the use of a dedicated line or network. In fact, in some cases, this can be the most convenient approach in fulfilling delivery-rate and security requirements.

End-to-end data flow model

In the context of this article, mission data consists of all the data from a given space mission that is of interest to its user community. Typically, this includes spacecraft data in raw or pre-processed forms and auxiliary data - the latter usually being the information that is needed by the user community in order to meaningfully interpret and process the spacecraft data.

Figure 1 depicts a typical end-to-end data flow model for sharing mission data with the user community. Note that it is not the intention to impose a system architecture, but rather to provide a model that should help to define the end-to-end system functionality. Spacecraft data is received by the Mission Control System (MCS) from the ground stations. In addition to being used for spacecraft monitoring and control, spacecraft data together with all the relevant auxiliary data are made available (pre-processed or not depending on the mission requirements) to the back-end of the MCS, the Mission Exploitation System (MES) and, in particular, to the Data Distribution System (DDS).

end-to-end data flow model
Figure 1.The end-to-end data flow model

A user community member who is interested in receiving mission data can prepare a request (see step 1, Fig. 1) on his local computer and submit it via the Internet to the DDS. Upon receipt, the DDS interprets the request, prepares the requested mission data and returns it to the user via the Internet (step 2a, Fig. 1) or via MDM (step 2b, Fig. 1). In the second case, some form of physical shipment (e.g. mail) is needed.

As a result, the user will have the requested mission data available locally on his computer, but will still need more information in order to interpret it. In an ideal situation - the one modelled here - all mission data delivered by the DDS is encoded in Standard Formatted Data Units (SFDU). As shown schematically in Figure 2, the SFDUs are recommendations for standard data structures developed by the CCSDS that provide a method for labelling data objects and linking them to a unique description. By reading the labels on the mission data, the so-called 'Authority and Description Identifier (ADID)', the user can query the Control Authority Office (CAO) system (step 3, Fig. 1) which will return the data descriptions (DD) corresponding to the labels (step 4, Fig. 1). The DD provides the information that allows unambiguous mission data decoding and interpretation.

relationship between SFDU-encode
Figure 2. The relationship between SFDU-encoded data and its data description

The existence of the CAO system might seem to bean excessive preoccupation at first glance, but it reflects a very realistic need, above all in the case of a long-duration mission or when access to the mission data is not restricted.

The Data Distribution System

The main task of the DDS is to deliver mission data to the user community. Generally speaking, the DDS provides the following two distinct services:

Any mission that requires a DDS needs to define and design it based on the volume of mission data to be shipped out, the size of the user community and the speed at which data needs to be provided to the user (Table 1). For example, a mission generating a high data volume and servicing a large user community usually cannot afford to only have the on-line service because of the costs associated with providing sufficiently performant network connectivity. It would then be advisable to restrict the on-line service to a small portion of the mission data and have the bulk delivered via an off-line mechanism. Mission critical information that requires timely delivery or summary information that allows the user community to establish its interest in receiving larger volumes of mission data are good candidates for on-line access.

Table 1. Indicative trade-offs between on-line and off-line data delivery services based on mission data volume and user community size


Mission Data Value

User Community Size

On-line DDS

Off-line DDS

High

High

Low

Low

High

Low

High

Low

X

 

X

X

X

X

 


On-line data delivery service
The requests for on-line service may include:

The way in which user requests are delivered to the DDS can vary considerably. For example, the Cluster scientific mission uses ASCII structured files transferred with the File Transfer Protocol (FTP), while XMM plans to adopt HyperText Transfer Protocol (HTTP) messages.

The protocol of the responses from the DDS to user requests also varies from project to project. For example, Cluster uses SFDU-encoded files transferred with FTP, while XMM plans to adopt HTTP messaging (although use of SFDU is currently under discussion).

Off-line data delivery service
The on-line and off-line data delivery services may be totally independent of one another although both may use the same mission data pool. For instance, this service may be wholly controlled by the DDS and not require any input from external users (as in Cluster) or be user-triggered via the Internet (as planned by XMM). In any case, once the DDS receives the order to produce one or more MDMs, it retrieves the data needed and starts the production process. As a general rule, the MDM and its format should comply with internationally recognised standards to ensure maximum portability across platforms. To fully exploit the potential of the Control Authority Office concept, it is highly desirable that the data on the MDM be SFDU-encoded, as is the case for Cluster and is currently under discussion for XMM (Table 2).

Table 2. Comparison between Cluster and XMM Data Distribution Systems


 

CLUSTER

XMM

Computer platform
User community
Mission data available on-line
On-line data delivery service

 

 

Protocol for on-line requests
Protocol for on-line responses
Off-line data delivery service

Mission Delivery Medium (MDM)
MDM standard

Data description registered
With CAO

VAX 4000-105A
Pre-defined list of about 20 users
Last 10 days of mission data
Yes.
- Requests for catalogue information
- Requests for on-line delivery of mission data for quick-look

 

ASCII structured files transfer with FTP
SFDU-encoded files transferred with FTP
Yes. Automatic daily production of
CD-ROMs for a selected distribution list
CD-ROM
SFDU-encoded CD-ROM compliant with
ISO 9660 level 1
Yes

Distributed system involving SUN and NT platforms
Entire astronomical community
Entire mission
Yes.
- Requests for catalogue information
- Requests for on-line delivery of mission data for quick-look
- Requests for off-line delivery of specific mission data
- Requests for user registration
HTTP messages
HTTP messages (SFDU TBD)
Yes. Automatic and on-demand (Internet)
Production of CD-ROMs
CD-ROM
CD-ROM compliant with ISO 9660 level 2
(SFDU TBD)
TBD


The Control Authority Office system

Background
With the large quantities of space science data being produced by today's space data systems, the format of the data is often not sufficiently well documented to allow users not directly involved with the systems to exploit the data. This is particularly the case where the data archiving is long-term and people with detailed knowledge of the data are no longer available.

To address this problem, the CCSDS has defined the CAO mechanism for ensuring that descriptions of data, are always easily accessible and available to users. It is part of the CCSDS SFDU concept explained above, which defines standardised methods for encapsulating data for exchange between providers and users (Fig. 2).

Clearly, the data descriptions could be packaged with the data, giving the advantage that the descriptions are immediately available to the user receiving it. The disadvantage, however, is that if the same format of data is sent repeatedly, for instance in case of spacecraft telemetry frames, then the same data descriptions need to be sent over and over, leading to unnecessary overhead. For this reason, the CAO concept handles the registration, revision and dissemination of data descriptions that do not accompany the data which they describe.

Each CCSDS member agency is obliged to nominate a primary Member Agency Control Authority Office (MACAO) with overall responsibility for the data descriptions registered at that agency. ESOC has been nominated as the primary MACAO for ESA.

The CCSDS has published two recommendations relating to Control Authority responsibilities and procedures. The top level recommendation is the 'Standard Formatted Data Units - Control Authority Procedures' Blue Book. This outlines the purpose of the Control Authority and the procedures that must be followed by the agencies conforming to the standard. It includes procedures for establishing a primary MACAO and any derivative MACAOs, the registration, dissemination and revision of data descriptions (the primary tasks), and the relevant reporting procedures.

The second recommendation of the CCSDS is the 'Standard Formatted Data Units - Control Authority Data Structures' Blue Book. This builds on the top level recommendation by presenting an automated method of interacting with a Control Authority. Standard SFDU structures are defined for carrying the information that is required for registration, dissemination and revision of data descriptions, plus standards for the representation of the necessary information. This recommendation relies upon the electronic transfer of the necessary information, it does not specify whether the transfer should be via physical magnetic media (e.g. disks or tapes), network transfer or an interactive form interface.

The ESA generic Control Authority Office system
As mentioned in the introduction, ESA took the lead within Panel 2 of CCSDS in developing a generic CAO system that would fulfil the CCSDS recommendations and that could be distributed as public domain software (free of charge) to any space agency that would require it. All of the software components necessary to run the ESA CAO system have therefore been selected so as not to require the payment of any licence fees. The system runs on a SUN/Solaris platform.

As defined by the CCSDS recommendations, the ESA CAO system supports the following services:

The current version (3.0) of the ESA CAO system (Fig. 3) supports two types of interfaces:

ESA control authority office system
Figure 3. The ESA Control Authority Office system components

The ESA CAO system is easy to install and customised configuration is possible. Each installation of the ESA CAO system can manage data descriptions for several MACAOs and has the ability to automatically forward requests for data descriptions that are not under its control to the appropriate CAO installation. The concept of having all CAO systems intercommunicating is shown in Figure 4.

inter-networked control authority office system
Figure 4. The concept of inter-networked Control Authority Office systems

Conclusion

This article has explained how greater use of the Internet could be exploited to cover some of the day-to-day needs of the users of space mission data. The appropriate balance between on-line and off-line data delivery services needs to be established based on the requirement drivers of the particular mission. This is a critical design decision in that it severely influences the data delivery performance and cost of any ground segment.

Unfortunately, no standard is currently imposed for the transfer of mission data to the user community via the on-line and off-line services. In this context, the SFDU mechanism would seem appropriate. Also, considering the services required from the DDS for several recent missions, it would seem that a consistent generalisation effort is possible and that the DDS could be a good candidate.

On the data description side, the ESA CAO system is already available, and its use for future missions therefore has a minimal cost impact.

Those readers interested in obtaining a copy of the ESA CAO system and CAO API software are invited to contact the author of this article.


About | Search | Feedback

Right Left Up Home ESA Bulletin Nr. 92.
Published November 1997.