Three hours to save Integral

As a result of the spacecraft turning, data were only reaching ground control patchily and the batteries were quickly discharging. With just a few hours of power left, it seemed possible that the 19-year-old mission could be lost.

The Integral Flight Control Team, together with Flight Dynamics and Ground Station Teams at ESA's ESOC mission control, teams at ESAC and Airbus Defence & Space, set to work. With quick thinking and ingenious solutions, they found the problem and rescued the mission.

What on Earth?

A Single Event Upset (SEU) occurs when a charged particle strikes a sensitive part of electrical equipment, causing a one-off ‘change of state’ that disrupts its functioning. These charged, ‘ionised’ particles often come from the Sun when it spews out matter and energy during solar flares or coronal mass ejections.

“I don't think that the SEU on this occasion was caused by our local, occasionally grumpy star. This strike happened on a day when no relevant space weather activity was observed,” explains Juha-Pekka Luntama, ESA’s Head of Space Weather.

“Based on a discussion with our colleagues in the Flight Control Team, it looks like that the anomaly was triggered by charged particles trapped in the radiation belts around Earth.”

The Van Allen radiation belts are two doughnut-shaped regions encircling Earth, where energetic charged particles are trapped inside Earth's magnetic field. Their properties vary according to solar activity and they represent a hazard to satellites and humans in space that pass through them. Because the lowest point of Integral’s orbit is now only 1500 km from Earth’s surface, the spacecraft passes through both radiation belts in its orbit.

‘Darmstadt, we have a problem’

Integral uses ‘reaction wheels’ – wheels that store energy as they spin – to subtly control the direction the spacecraft points in without the need of thrusters.

Suddenly, one of these reaction wheels stopped and, because of the law of conservation of energy, that turning force previously in the wheel had to go somewhere else – the entire spacecraft. The spacecraft began to spin, triggering an Emergency Safe Attitude Mode which unfortunately, due to a previous failure, was no longer reliable and did not manage to stabilise the mission.

The reaction wheel was reactivated by teams on the ground, but the spacecraft kept spinning at an average rate of about 17 degrees per minute (roughly one rotation every 21 minutes), as well as wobbling unpredictably about its axes. This may not sound like much, but the spacecraft was rotating at five times its maximum when under control.

“The data coming down from Integral was choppy, coming in for short periods due to it spinning. This made analysis even harder,” explains Richard Southworth, Operations Manager for the mission.

“The batteries were discharging, as there were only short charging periods when the panels briefly faced the Sun.”

The first challenge was to decrease Integral’s energy consumption to buy more time. First estimates of the charge remaining before blackout and the loss of the satellite was just three hours. Step by step, by turning off various instruments and non-critical components, this increased to more than six hours. Next step – stop the spinning.

With support from industry experts, the team at ESOC analysed the state of the reaction wheels, coming up with a series of commands to change their speed and brake the spinning satellite. By late afternoon, the commands were sent and immediately showed success, but another three long hours passed before the satellite was fully under control and out of immediate danger.

Integral’s ‘Apollo 13’ moment

“Everyone breathed a huge sigh of relief. This was very close, and we were immensely relieved to get the spacecraft out of this ‘near-death’ experience,” recalls Andreas Rudolph, Head of the Astronomy Missions Division in ESOC's Mission Operations Department.

“Most of the Control Team were working from home at this point – I was following operations from the train! – and worked until four in the morning to get the spacecraft fully stable, back into position and facing the Sun to recharge its batteries.”

Unfortunately, a few hours later as the team reconvened to discuss the next steps, the spacecraft once again started to rotate, its reaction wheels again turning at high speed. The reason for this is still not completely understood but is thought to be associated with a ‘star tracker occultation’ or ‘blinding’ which wasn’t handled correctly by the satellite’s control systems – effectively when Earth gets in the way of the spacecraft’s view of the stars, which it uses to orient itself.

The team repeated the previous days steps to stabilise the spacecraft and return to a Sun pointing position, this time without getting in the way of the star trackers. The recovery took just a couple of hours, putting in to practice the lessons learnt from the first time.

Integral has since remained under control, and from 27 September all systems are back online. Since 1 October, after an extended checkout, its instruments are back observing the high energy Universe.

One of the first targets for Integral will be to observe massive stars in the Orion region, and study the impact on their surroundings when they go supernova.

“We are also back to ‘target of opportunity’ observations, which means that Integral is again reacting quickly to study unexpected explosive events in the Universe,” says Erik Kuulkers, ESA’s Project Scientist for Integral.

A problem of thrust

It’s not the first time this almost 20-year-old mission gave the control team at ESA’s ESOC Operations Centre a scare. Last year, Integral fired its thrusters for possibly the last planned time, after a failure with its propulsion system.

It’s this deficient propulsion system that meant a normally rectifying Safe Mode was ineffective on this occasion. With the mode now disabled, the Control Team are working on a new automatic rescue sequence that should mimic many of the operations carried out after this anomaly, only much faster.

When the propulsion system failed, the team realised they would have to learn to manoeuvre the four-tonne satellite using its highly sensitive reaction wheels alone, to dump energy at regular periods and counteract forces on the spacecraft, including the gentle shove from the Sun’s light. It was a solution that had never been tried before.

“I didn’t believe it was possible at first. We checked with our flight dynamics colleagues and the theory indicated it would work. After doing a simulation, we tested it on the spacecraft. It worked,” explains Richard.

“Thanks to our quick-witted team and the help of experts from across industry, Integral lives on. Almost two decades old, it is far outliving expectations for what was meant to be a five-year mission.”

Listen to Richard recount the story of Integral’s previous rescue and upgrade and the amazing teamwork that saved the mission, in the latest episode of the ESA Explores Space Operations podcast.