Human Factors and Redundancy on Apollo 10 and Skylab 4
The Apollo and Skylab Programs each suffered major setbacks and losses. These events, such as the Apollo 1 fire, the Apollo 13 oxygen tank explosion, and the Skylab 1 micrometeoroid shield loss, continue to serve as lessons and reminders to engineers and managers at NASA and in the expanding private spaceflight industry. As the major incidents of these heritage Programs linger in memory, multiple lesser known crises were averted during the 21 missions that utilized the Apollo spacecraft. During Apollo 10 and Skylab 4, crews suffered from human factors related incidents that were remedied by engineered redundancies and excellent operational knowledge of the spacecraft.
Figure 1. The Apollo LM, nicknamed “Snoopy,” on approach to the CSM “Charlie Brown,” following the staging sequence. Source: NASA
Apollo 10 launched on May 18, 1969. Its main objective was to carry out manned Command/Service Module (CSM) and Lunar Module (LM) operations in lunar orbit in preparation for the Apollo 11 lunar landing two months later.
The crew of Apollo 10 completed translunar injection and the transposition, docking and extraction maneuver. Three days after launch, the Commander (CDR) and Lunar Module Pilot (LMP) entered the LM, fired the descent engine and executed the descent orbit insertion maneuver. Then, at the altitude of 50,000 feet, the two-member LM crew tested the landing radar. During a lunar landing mission, like Apollo 11, the LM crew would have initiated a powered lunar descent; however, Apollo 10 was only equipped with enough fuel to survey the Apollo 11 landing site in the Sea of Tranquility.
Control Loss at Staging
During the last LM pass, the crew donned helmets and gloves for the “staging” maneuver — when the crew would jettison the LM decent engine. At 28 seconds to staging, the LM attitude indicator showed a slight yaw rate from the commanded attitude. Telemetry suggested they might have an electrical anomaly, so the CDR began to troubleshoot the problem. It was difficult to reach the right switch with helmet and gloves on and the CDR inadvertently cycled the Abort Guidance System (AGS) mode control switch from the “ATT HOLD” (attitude hold) position to “AGS AUTO.” The LM immediately entered a rapid end-over-end roll as the AGS automatically searched for the CSM five seconds before staging. The roll continued for eight seconds as the altitude indicator moved into the red zone and the danger of losing the inertial platform rose. The CDR grabbed the hand controller, switched the LM to manual flight, jettisoned the descent stage and stabilized the ascent stage.
Figure 2. Comparison of actual (left) and planned (right) LM attitudes leading up to and following the staging sequence. Source: NASA
The LM attitude indicator “GIMBAL LOCK” light came on. The inertial guidance platform was in danger of losing its reference data which would cause the loss of LM attitude indication to the crew. Crew inspection showed that inertial platform was still stable however, and the primary guidance system was still usable.
Fuel Cell Pump Failure
The LM crew docked with the CSM and jettisoned the LM. While the CSM was still in lunar orbit a Caution & Warning (C&W) alarm sounded. Fuel Cell 1 AC circuit breaker had tripped, due to a short in the hydrogen pump, which caused a loss of Fuel Cell 1. Out of contact with Earth, the crew proceeded with the mission, following procedures to minimize non-essential electrical loads on the two remaining redundant fuel cells. The crew took photographs of the lunar surface until they emerged from the night side. Once in contact with Mission Control, their actions to limit non-essential electrical loads were validated. Had the crew aborted the mission, splashdown location and timetable shifts would have placed them well out of range of assigned recovery assets.
The Command Module Pilot (CMP) jokingly stated “I bet when we get our next loss of signal another fuel cell’s going to fail.” The Fuel Cell 2 Caution Light proceeded to come on, followed by a Warning Light: the fuel cell’s condenser exhaust temperature was cycling between its high and low limits.
The crew immediately shut off automatic fuel cell heat to reduce electric load. Fuel Cell 2 continued to provide power while the crew manually controlled the heaters and monitored fuel cell skin temperature.
During the control loss at staging, the CDR’s attention was focused on reacting to an LM electrical anomaly while involved in procedures for LM ascent/descent separation. Additionally, the crew had limited reach, dexterity, and visibility while fully suited, with helmet and gloves on.
Apollo 10 was a successful “dress rehearsal” for the Apollo 11 lunar landing. The mission also set records for the highest speed attained by a manned vehicle during return to Earth (24,791 miles per hour) and for the furthest humans have travelled from home in Houston (254,109 miles) when on orbit on the far side of the moon while Houston was the farthest on Earth’s rotation.
The last of the Skylab missions, Skylab 4, launched on November 16, 1973. The mission completed 1,214 Earth orbits, had four Extravehicular Activities (EVAs) totaling more than 22 hours, and lasted 84 days. Although the crew set a record for length of time in orbit, Skylab 4 was the first mission for each of the three crewmembers. The rookie Skylab 4 crew experienced issues while attempting to accomplish the ambitious workload initially outlined by ground control.
Figure 3. The SCS CBs that were incorrectly pulled are located two rows above the SPS CBs, which are highlighted in violet). Source: NASA
The crew was troubleshooting an issue with the Command Module (CM) Reaction Control System (RCS) System Ring 2 while involved in procedures for CM/SM separation. Prior to SM separation, procedures dictated that the crew pull four Service Propulsion System (SPS) circuit breakers (CBs) to “deadface” or de-mate the power connection to the unnecessary SPS pitch and yaw gimbal motors.
During deorbit, after the CM/SM separation, the crew noticed that the Stabilization Control System (SCS) and RCS system was not automatically maneuvering the CM from apex forward to entry attitude (aft heat-shield forward). If the CM attempted entry in the apex forward configuration, it would result in the loss of spacecraft and crew. The CDR quickly decided to switch RCS control to “direct,” allowing the Rotation Hand Controller to directly power the RCS solenoids and maneuvered to the proper entry attitude.
Figure 4. Color-enhanced ultraviolet exposure of a colossal solar eruption photographed during the Skylab 4 mission. Source: NASA
Post flight, the crew stated that during the time critical preparation for SM separation, they inadvertently pulled SCS Pitch and Yaw CBs instead of the SPS pitch and yaw CBs. The two sets of CBs were located two rows apart from each other on the CB panel and were similarly labeled.
The successful splashdown of the Skylab 4 crew marked the end of Skylab missions. Although the Skylab missions were troubled early on with the loss of the station’s micrometeorite shield and one of its primary solar arrays, Skylab logged approximately 2,000 hours of experiments and 173,000 film images that would have been impossible for unmanned systems to duplicate. Skylab also paved the way for NASA’s research into long-term human space habitability.
Relevance to NASA
During Apollo 10, the CDR saved the vehicle and crew by using a direct manual redundant backup to the automated LM RCS. Later in the mission, two-fault tolerant CSM Fuel Cells assured mission success, as the CSM could safely returned with one fuel cell powered down, and Apollo 10 with two fuel cells powered down.
Similar to Apollo 10, during Skylab 4, the manual redundant backup to auto RCS control was direct which saved this vehicle and crew.
Critical automatic functions should have a manual or unlike redundancy backup. Consumables, like electrical power required for crew safety, should have additional levels of redundancy.
Figure 5. Skylab as viewed from the departing Apollo 4 crew. Source: NASA
In both missions, the CDRs’ intimate knowledge of system operations led to quick recoveries and assured the safety of the crew and spacecraft. Extensive crew knowledge of nominal and off-nominal systems operation is essential when immediate action is required and communications with Earth are delayed or blocked. Although more than three months had elapsed since the crew had flown the CSM, the Skylab 4 crew was able to rely on their training to quickly regain manual control during the RSC yaw and pitch loss. Similarly, the Apollo 10 CDR’s reaction jettison the descent stage and manually stabilized the ascent stage to save the mission and crew.
The CDRs cognitive performance under high-stress conditions is a testament to NASA astronaut training during the Apollo-era. However, both short-term and long-term memory is subject to degradation. Training discussions can be done inflight to regain and maintain proficiency during long duration missions and refresher simulations can be conducted before time-critical operations.
Questions for Discussion
- Are multiple safeguards available during early operation?
- Are contingency plans for on-orbit anomalies adequate?
- Are interfaces adequately designed for operators who may be encumbered, fatigued, or stressed?
- How will the spacecraft operate under low power constraints?
Godwin, Robert, ed. “Apollo 10: The NASA Mission Reports.” Burlington, Ontario: Apogee Books, 1971.
JSC-08809, Skylab 4 Technical Crew Debrief. February 22, 1974. http://spaceflight.nasa.gov/outreach/SignificantIncidents/ assets/skylab-4-technical-debrief-1-2.pdf NASA. Accessed: Aug. 10, 2016.
NASA JSC Oral History Project, Gary W. Johnson Oral History. May 3, 2010. http://www.jsc.nasa.gov/history/oral_histories/ JohnsonGW/johnsongw.htm. NASA. Accessed: July 22, 2016.
Significant Incidents & Close Calls in Human Spaceflight. http:// spaceflight.nasa.gov/outreach/SignificantIncidents/training-and-human-factors.html. NASA. Accessed: Aug. 9, 2016.
SP-4208 Living and Working in Space: A History of Skylab. http:// history.nasa.gov/SP-4208/contents.htm. NASA. Accessed: July 22, 2016.
Stafford, Thomas P and Robert Godwin, ed. “We Have Capture.” Washington D. C.: Smithsonian Institution Press, 2002.
Woods, David, Robin Wheeler and Ian Roberts. The Apollo 10 Flight Journal. http://history.nasa.gov/ap10fj/index.htm. NASA. Accessed: Aug. 10, 2016.
Responsible NASA Official: Steve Lilley
Special thanks to Gary Johnson for his contributions to this study.
This is an internal NASA safety awareness training document based on information available in the public domain. The findings, proximate causes and contributing factors identified in this case study do not necessarily represent those of the agency. Sections of this case study were derived from multiple sources listed under References. Any misrepresentation or improper use of source material is unintentional.