United 232: the word was independent

Forty-four minutes

On 19 July 1989 United Airlines flight 232, a McDonnell Douglas DC-10-10 delivered in 1971, left Denver for Chicago at 14:09 with 285 passengers and eleven crew on board. The first officer was flying. The climb to 37,000 feet was uneventful. The captain had 29,967 hours with the airline, 7,190 of them on the DC-10.

One hour and seven minutes after takeoff, the crew heard a loud bang.

1. 15:16:10A bang, then vibration and shuddering. The engine instruments showed the No. 2 engine, mounted in the tail, had failed. The captain called for the shutdown checklist. While running it, the flight engineer saw that the hydraulic pressure and quantity gauges for all three systems read zero. The first officer reported he could not control the airplane as it entered a right descending turn.
2. 15:20The crew asked Minneapolis Center for emergency assistance and vectors to the nearest airport.
3. 15:22The controller offered Sioux City. The crew accepted.
4. 15:29A United DC-10 check airman, traveling off duty in first class, arrived in the cockpit. The captain gave him the throttles so that two pilots could stay on the controls.
5. 15:42The flight engineer was sent back to look at the tail. He reported damage to both horizontal stabilizers.
6. 15:49Landing gear extended by the alternate procedure. This was the last transmission recorded on the airline's company channel.
7. 16:00Touchdown on the threshold of runway 22, a closed runway the controller had told them was 6,600 feet long, at an average of 215 knots and 1,620 feet per minute in the last twenty seconds. The right wing dropped, struck first, and the airplane broke up and burned.

Of the 296 people on board, 110 passengers and one flight attendant were killed, and one more passenger died of his injuries thirty-one days later. Forty-seven were seriously injured, 125 had minor injuries, and thirteen walked away unhurt. The investigators found that 35 of the dead had died of smoke inhalation, 24 of them with no blunt force injuries at all. The airplane, valued at $21 million, was destroyed.

The crew flew it for forty-four minutes with no working flight controls, using nothing but the throttles of the two wing engines, and got it to a runway. The Safety Board's finding on that is unusually direct: the airplane was marginally flyable and a safe landing was virtually impossible. We will come back to why that is a design finding rather than a piloting one.

What independent meant

The DC-10's flight controls are hydraulic and only hydraulic. There is no cable, no manual reversion, no way to move a surface without fluid under pressure. The report states it plainly: at least one hydraulic system must remain intact, with fluid present and the ability to hold pressure, for continued flight and landing. The airplane was certified on the strength of having three.

Each of the three systems is driven by its own engine, with a primary and a reserve pump. Every primary and secondary control surface is powered by two of the three, so that losing any one system, or any two, leaves every axis controllable. Reversible motor pumps can move power from one system to another without connecting the fluid. An air-driven generator can be dropped into the airstream to power an electric pump on the No. 3 system. The rudder actuators sit on closed loops so that damage at the rudder cannot drain the systems that feed them.

And the lines were kept apart on purpose. The report describes it: the No. 1 lines run along the left side of the fuselage to the tail, the No. 3 lines along the right side, the No. 2 lines come from the center engine along the rear spars. No. 2 is never routed forward of the rear wing spar, to keep it clear of fragments from the wing engines. No. 3 is never routed aft of the inboard elevator actuators in the tail, to keep it clear of fragments from the tail engine. These are design precautions against exactly the hazard that killed the airplane. Somebody drew the fragment cones and moved the pipes.

Independent was not a word anyone used loosely. It was three separate fluids, three separate pumps, three separate engines, physically separated routing, and a regulation, 14 CFR 25.901, requiring that no single failure or probable combination of failures would jeopardize the airplane. The design met it. The FAA agreed that it did.

Where they met

The three systems meet in the horizontal stabilizer, because the elevators live there and each elevator is powered by two of the three. The right inboard elevator takes systems 1 and 3. The left inboard takes 2 and 3. The outboard elevators take 1 and 2. To reach those actuators, lines from all three systems have to cross the stabilizer. And the stabilizer sits directly beneath the No. 2 engine.

The reconstruction of the tail is the center of the report. A photograph taken from the ground as the airplane came in shows a section of the right stabilizer already missing in flight. That section carried the No. 1 lines feeding the right elevator actuators. A piece of No. 1 tubing with a tee fitting attached was later picked up off the runway, bent and punctured, with titanium alloy traces on it. The No. 3 pressure line was found severed in the inboard right stabilizer, with a gap of about two and a half inches, and the return line had been driven through the structure and torn from its fitting. The material on the cut ends was titanium alloy. The hole where the object had entered through the top skin was 5.5 inches by 2.25 inches, roughly rectangular, and did not match any piece of the disk. It matched the base platform of a fan blade.

The No. 2 system did not need a fragment. Its supply and return hoses, still clamped together, were found in a field near Alta, Iowa, attached to the No. 2 engine-driven pump. The forces of the engine failure itself had torn them away. All three reservoirs were found empty. The finding is one sentence:

No. 2 engine fragments severed the No. 1 and No. 3 hydraulic system lines, and the forces of the engine failure fractured the No. 2 hydraulic system, rendering the airplane's three hydraulic-powered flight control systems inoperative.

NTSB AAR-90/06, Finding 5

Note the routing rule again. No. 3 was kept out of the tail aft of the inboard elevator actuators. It was cut in the inboard part of the right stabilizer, on its way to the actuators it was allowed to reach, because the rule had been drawn around a fragment cone that turned out to be the wrong shape.

Extremely remote

Here is the certification record, as the report sets it out, and it is worth reading as a sequence.

The DC-10-10 was certified against Part 25 as it stood in February 1965, with amendments through 22. The regulation on rotor failure at that time, 25.903(d), was in the middle of being revised. The FAA judged that the existing text did not contain adequate or appropriate standards for the DC-10, and on 7 January 1970 imposed Special Condition 25-18-WE-7 instead. It required the airplane to incorporate design features to minimize hazardous damage in the event of an engine rotor failure.

On 1 July 1970 Douglas supplied information to the FAA indicating that the powerplants and associated systems were isolated and arranged so that the probability of the failure of any one engine or system adversely affecting the others was extremely remote, and that the hydraulic system design considerations demonstrated compliance. On 17 July 1970, sixteen days later, the FAA responded that its review was complete and the requirements were satisfied.

That is the whole exchange on the question that decided this accident. Now the sequence continues.

In November 1975 the FAA issued Order 8110.11, Design Considerations for Minimizing Damage Caused by Uncontained Aircraft Turbine Engine Rotor Failures. It contained fragment spread angles and suggested shields or deflectors. It was distributed internally to FAA offices. In 1982 the Safety Board recommended published guidance. In March 1988 the FAA issued Advisory Circular 20-128, which defined the spread angles, the predicted piece sizes and the energy levels an airframe designer should assume, and proposed that critical systems be located outside the debris area or shielded. The report's own summary:

For almost 20 years, and obviously during the development period of the majority of the wide-body fleet, a recognized interpretation of the regulations concerning hazards related to uncontained engine failures was not published by the FAA.

NTSB AAR-90/06, section 2.7.2

And when the guidance did arrive, it assumed a disk breaking into three pieces with one third penetrating the airplane. The disk on flight 232 was recovered three months later from cornfields near Alta, in two pieces that together made up the whole of it, less one dovetail post. The Board found that the dispersion pattern and the energy levels in this accident were inconsistent with the predictions of the circular that had finally been published the year before.

The engine side of the record is shorter. Part 33 required the engine case to contain a released fan blade. It did not require containment of a fan disk, and no advisory material asked for it. The containment ring around the CF6-6 fan is a stainless steel hoop designed to absorb the energy of one released blade and the adjacent damage. The smaller piece of the disk burst straight through it. Nothing in the engine's certification was broken by this accident, because a disk had never been in scope. Nothing in the airplane's was either. The gap was between them.

The disk

The design story is the reason we chose this accident, but the probable cause begins somewhere else, and leaving it out would be dishonest. Fan disk serial MPO 00385 was made at General Electric's Evendale plant between September and December 1971. The titanium it was forged from carried a hard alpha inclusion, a region where the molten metal had absorbed too much nitrogen and become brittle. The defect was in the bore, which the report calls a critical area of the disk. It was not found by the ultrasonic and macroetch inspections during manufacture, and the Board concluded that the cavity at the fatigue origin was most likely created during final machining or shot peening, after those inspections had already been done. Had the part been macroetched in its final shape, the flaw would have been apparent.

The crack began on the disk's first exposures to full thrust and grew for seventeen years. In that time the disk was removed six times for fluorescent penetrant inspection and accepted six times. At the last of them, 760 cycles before the accident, the crack was about half an inch long on the bore surface. The investigators proved it was there: the fracture face carried a discoloration marking the crack's size at that inspection, and the washings contained the phosphate compounds of penetrant fluid. The fluid had gone into the crack. The inspector had not seen the indication. The disk was returned to service as flaw-free, installed in the engine that went into the No. 2 position on N1819U, and ran 760 more cycles.

The Board also found that General Electric's manufacturing records for the disk were incomplete, that two disks had carried the same serial number, and that the records system did not assure accurate traceability of rotating parts. It spent some effort establishing that the two disks had not been switched. They had not. But a company that cannot say with confidence which of two identically numbered parts went where has a records problem, and every analysis downstream inherits it.

The probable cause names the inspection first: inadequate consideration of human factors in the inspection and quality control at the overhaul facility, which failed to detect a crack from a previously undetected metallurgical defect. Then the second half, which is the half this series is about:

The subsequent catastrophic disintegration of the disk resulted in the liberation of debris in a pattern of distribution and with energy levels that exceeded the level of protection provided by design features of the hydraulic systems that operate the DC-10's flight controls.

NTSB AAR-90/06, Probable Cause

Exceeded the level of protection provided by design features. The protection was real. It was designed, drawn, built and certified. It was sized for a threat that was smaller than the one that arrived.

One member of the Board, Jim Burnett, would have gone further and filed a dissent saying so. His probable cause named three failures in order: the manufacture of a defective disk, the failure to find the crack, and the failure of the airframe design to account for a random release of engine fragments, with the FAA's certification process as a contributing cause for not requiring it. The event, he wrote, was foreseeable even though remote, and neither Douglas nor the FAA was entitled to dismiss a possible rotor failure as remote when reasonable and feasible steps could have been taken to minimize the damage. The proof, he said, was the retrofit that followed. The majority put the same facts in the second sentence of the cause rather than the first. The engineering reading is the same either way.

The analysis nobody was required to do

The report's section on design philosophy is where the system safety lesson lives, and it is more precise than the usual account of this accident.

The Board did not fault the number of hydraulic systems. It looked at three-system and four-system designs across the wide-body fleet and could find no inherent safety advantage in adding more. Three independent systems, it said, is not unique and not inadequate. The problem was not the count. It was what independent had been checked against.

It then described the gap. An engine manufacturer certifies an engine to Part 33 without regard to any particular airframe. The same engine goes on several airplanes from several companies, wing-mounted, fuselage-mounted, tail-mounted, in different numbers. Each installation carries its own hazards. The Board's words are that these differences require specific system safety assessments that are not currently explicitly required. The airframe designer needs fragment sizes and energies that only the engine maker can supply, and the Board noted, carefully, that a company selling an engine may underestimate the potential for failure, and a company buying one may not have the data to estimate the effect on the airframe.

In retrospect, the Board said, the potential for hydraulic damage from random engine debris should have been given more consideration in the original design, and Douglas should have better protected the critical systems. Additional design precautions could have been taken if the distribution pattern and the fragment energy levels had been predicted.

Read that as a systems engineer rather than as a historian. The analysis that was missing has a name today. It is the particular risks assessment, and the uncontained engine rotor is its first and most classical case. Its whole purpose is to take an event that sits outside the system boundary, a disk fragment, a tire burst, a bird, a fire, and ask what it does to every system that happens to be in its path, regardless of how independent those systems are of each other on paper. The DC-10's hydraulic routing was a particular risks assessment done by hand, against a fragment cone, nineteen years before the guidance existed. It was a good one. It was wrong about the cone, and nothing downstream could tell.

Where the tool sits in this

Independence is a geometric claim before it is a probabilistic one. Three systems are independent against an uncontained rotor only if their lines do not share the region the rotor can reach, and that is a question about routing and zones, not about failure rates. On the DC-10 the answer was yes everywhere except a few feet of the horizontal stabilizer, and that is where the airplane was lost.

That is the reason our platform treats zones and particular risks as part of the same model as the systems, rather than as separate documents. Every line, actuator and reservoir belongs to a zone. Every particular risk carries a footprint, and for a rotor burst that footprint comes from the engine installation, not from a default. When two systems that an analysis calls independent both route through a zone inside that footprint, the dependency is flagged in the same pass as the failure rates, deterministically, with the same inputs producing the same result every time, and with the finding traceable to the lines and the zone that caused it. Change the fragment model, as AC 20-128 changed it in 1988, and every independence claim that leaned on the old cone re-evaluates. Nobody has to remember which drawings it touched.

None of this requires a blank sheet. The engineer starts from a populated analysis that already knows what runs where, and spends their time on the judgment, which in 1970 would have been whether a fragment cone drawn around a blade was the right cone to draw around a disk.

Nobody could have landed it

After the accident the Board put DC-10 line captains, check airmen and production test pilots into a simulator programmed with the accident airplane's recorded behavior and asked them to fly the profile with the wing engines alone. Asymmetric thrust gave some roll control. Thrust changes gave a limited and imprecise effect on pitch, with the airplane oscillating through a phugoid of about sixty seconds that could not be damped. Airspeed could not be controlled directly at all. The report's conclusion is that the maneuver was not trainable, that landing at a chosen point and speed was a highly random event, and that the airplane, although flyable, could not have been successfully landed. The crew's performance, it said, was highly commendable and greatly exceeded reasonable expectations.

Keep hold of one sentence from the same section, because it is the operational face of the design assumption:

Douglas Aircraft Company, the FAA, and UAL considered the total loss of hydraulic-powered flight controls so remote as to negate any requirement for an appropriate procedure to counter such a situation.

NTSB AAR-90/06, section 2.3

There was no checklist because the analysis said there did not need to be one. The most demanding thing the crew had ever practiced was the loss of two systems out of three, with the third still moving the controls. That is what extremely remote looks like when it is written into the training syllabus.

The historical review in the report lists what else had happened by then: a 747 at San Francisco in 1971 that lost three of four systems on takeoff and landed on the fourth; an L-1011 at Colts Neck in 1981 whose tail engine fan escaped and took three of four systems, and which also landed on the fourth; and a 747 at Mount Ogura in 1985 that lost all four and crashed with more than 500 dead. Each is a data point about where independent systems meet. None of them was treated as a reason to look at the DC-10 tail again.

What they did about it

Two months after the accident Douglas announced a change to the DC-10's hydraulic system, and the FAA mandated it in Airworthiness Directive 90-13-07, effective 20 July 1990, with a compliance deadline of a year. It is a small installation and it is worth understanding, because it is what an engineering answer to a dependency looks like.

An electrically operated shutoff valve was added to the No. 3 system supply line, forward of the horizontal stabilizer, with a check valve in the return. A sensor in the No. 3 reservoir closes the valve automatically if the fluid level drops below a preset point, and a light tells the crew it has done so. The effect is that if the tail is breached, the No. 3 system loses only the part of itself that runs through the tail. The rest of it keeps its fluid, and with it the stabilizer trim at half rate, three ailerons, the slats, spoilers, brakes, nose wheel steering, landing gear and lower rudder. Douglas also offered flow-limiting fuses as an interim measure. The MD-11 was built with the shutoff valve from the start.

Notice what the fix does not do. It does not add a fourth system, which the Board had already said would not help. It does not re-route the lines, which the geometry of elevators and a tail engine does not allow. It accepts that the three systems are not independent in that one zone and makes sure that losing the zone does not mean losing the airplane. It converts a shared point of failure into a bounded one. That is the correct response to a dependency the analysis missed, and it is available on any program at the design stage, at a fraction of the cost, if the dependency is found there.

The Board's recommendations went wider than the DC-10. It asked the FAA to conduct system safety reviews of currently certificated aircraft for the redundancy and protection of power sources for flight and engine controls. It asked for the fragment sizes and energies from this accident to go into a revision of AC 20-128, with the engine manufacturer required to provide them and the airframe manufacturer required to consider them. It asked for a central database of uncontained rotor events, having discovered that the most recent available industry data stopped in 1983. And it asked for research into backup flight control systems using a different source of motive power, because manual reversion on an airplane this size is not practical.

Three questions worth asking on Monday

This accident is a geometry problem that was certified as a probability problem. The questions it leaves are about the difference.

One. Pick any two systems your architecture calls independent, and draw where they come within a few feet of each other. Not on the schematic, on the airplane. If you cannot do it from your analysis without going to the installation drawings, the independence claim is not in your analysis. It is in somebody's head.

Two. For every particular risk in your assessment, name who gave you the footprint and write down what they assumed. The DC-10's designers drew a cone. The guidance that arrived eighteen years later drew a different one, and the accident drew a third. Your rotor burst model came from somewhere. If it came from a default, say so, because a reviewer will ask.

Three. Find every place your safety case uses the words extremely remote, and sort them into the ones with a calculation behind them and the ones with an arrangement drawing behind them. Both can be right. Only the second kind stops being right when the fragment gets bigger, and nothing in a probability will tell you when that has happened.

If those are uncomfortable, that is the finding. The Board's words were that the potential for this should have been given more consideration in the original design. The original design is the only place it is cheap.

What we are and are not claiming

We are not claiming that software would have saved flight 232. The analysis that mattered was done in 1970 with the tools of 1970, and the guidance that would have changed its inputs did not exist for another eighteen years. No model fixes a missing input. We are also not claiming the crew could have done anything differently. The Board settled that, in the simulator, with the people best placed to know.

The narrower claim is the one we will stand behind. The three hydraulic systems were independent against every hazard the analysis had been told to consider, and were not independent against the one that happened, and the reason nobody saw that is that independence was recorded as a word rather than as a geometry. Where the lines ran, what zone they shared, and what could reach that zone were three facts held in three places, and the analysis that would have put them together was, in the Board's phrase, not explicitly required. On any program being designed today it is required, and the tooling exists to do it as a property of the model rather than as a drawing exercise someone remembers to repeat.

The word was independent. That is the part worth carrying away. A claim of independence is only as good as the list of things it was tested against, and the list is never finished.

We build the tooling for this, and we do the analysis itself. If you are building an aircraft and want to know whether your independence claims survive your particular risks, we would like to look at it with you.

Sources. All factual material is from the published public record.

Aircraft Accident Report: United Airlines Flight 232, McDonnell Douglas DC-10-10, Sioux Gateway Airport, Sioux City, Iowa, July 19, 1989. National Transportation Safety Board, NTSB/AAR-90/06, adopted 1 November 1990. Sections 1.1, 1.2, 1.6, 1.12.4, 1.16, 1.17.6, 1.17.8, 1.17.9, 1.17.10, 2.3, 2.4, 2.6, 2.7, 3.1 and 3.2.

14 CFR 25.901 and 25.903, 14 CFR 33.19 and 33.27, Special Condition 25-18-WE-7, FAA Order 8110.11, FAA Advisory Circular 20-128, and FAA Airworthiness Directive 90-13-07, each as quoted or described in the report above.

Photographs: FAA Lessons Learned from Transport Airplane Accidents, United Airlines Flight 232, McDonnell Douglas DC-10-10, N1819U (faa.gov), from the NTSB investigation.

Quotations are short extracts attributed to the report. Every figure, time, dimension and date in this article is as stated in the report. Where the report gives no figure, none is given here. Where the report is internally inconsistent about the month of the final inspection, we have given the cycle count instead. The diagrams are our own and are schematic; the photographs are from the government record and credited where they appear.