*** What's wrong with the 787? ***

[LakeCleElum]

I spent the weekend with 33 friends on an overnight snowmobile trip. We had time to drink and talk....Many worked for Boeing, both as Engineers and Machinist.....

Talk was the "inside story" on the problems with the 787 Battery issue. 2 educated opinions:

1) All the cabon composite creates friction that builds up static electricity......Creates problems with excessive battery volatge?

2) Engineering has been out-sourced......Certain countries (Italy) have totally different standards and does not mesh well with the product the Boeing Engineers in the US have done.......

Just the rumors.....Any opinions? Jeff Higgins, anything you can repeat?

[HardDrive]

1) Seems far fetched. You would think that in long term stress tests, this behavior would have been observed.

2) Far more likely.

I think there is something wrong with the charging system.

[GWN7]

The other thread on this mentioned the batteries are made in Japan (or do they outsource them from China?)

[slodave]

http://www.nytimes.com/2013/01/29/business/global/boeing-787-batteries-pass-inspection-in-japan.html?_r=0

The Li-on batteries are supposed to be a hole lot more stable than Li-po's, but they still do fail (or combust).

[Hydrocket]

I have no evidence about the 787...just some anectodotal evidence from my own experience.

I had a carbon tail piece on my race bike. The battery is located underneath the tail. It didn't actually touch the tail, but was located underneath it. My tail caught on fire on 2 seperate occasions, many months apart. No rhyme or reason...

[KFC911]

No energizer bunny

[tevake]

Hydrorocket, did you look into just what you had eaten on those days, Just sayin

Cheers Richard

[onewhippedpuppy]

1. No, the fuselage will have established grounding paths and static wicks to dissipate built up charge. Those are the little straw looking things that protrude from the trailing edge of the wing and tail.

2. Very possible.

It's an airplane of firsts for Boeing. That always brings issues. First built by outsourcing the design and production of major component sub-assemblies. For instance, Spirit Aerosystems builds the nose section. In theory, the completed nose section simply bolts to the fuselage (designed by another supplier), the harnesses connect, and that's it. See potential for issues there? I've worked on a similarly structured program and it was a nightmare. Very difficult to coordinate suppliers around the world that are responsible for complex and interconnected parts. One instance in which outsourcing typically is not worth the trouble, at a minimum Boeing should have done the design work in-house and outsourced manufacturing (IMHO).

It's also Boeing's first carbon composite airplane, which is a major departure from metal planes. Their first airplane that relies on the electrical system for deice and other functions that are normally taken care of by engine bleed air. Any time you do something for the first time there are going to be issues, particularly in something as complex as an airplane. It's just a shame for Boeing that the issues are so public. I'm very curious how these problems did not arise during the flight test program. Schedule pressure on a program that was already years behind was probably a contributing factor.

[Jeff Higgins]

You guys know I can't say anything specific. I can, however, post something that is already in the public domain.

Steve Denning wrote this for Forbes magazine. It is the most accurate assesment I've seen to date. It speaks to the climate at Boeing, not specific technical problems on the 787. It's the best I can do right now:

My article, The Boeing Debacle: Seven Lessons That Every CEO Must Learn, elicited spirited conversation. Several commentators noted that, in addition to the general lessons, Boeing made specific errors in the way it handled outsourcing and offshoring. Let’s take a closer look at those specifics.
Boeing enthusiastically embraced outsourcing, both locally and internationally, as a way of lowering costs and accelerating development. The approach was intended to“reduce the 787′s development time from six to four years and development cost from $10 to $6 billion.”
The end result was the opposite. The project is billions of dollars over budget and three years behind schedule. “We spent a lot more money,” Jim Albaugh, Chief of Commercial Airplanes at Boeing, explained in January 2011, “in trying to recover than we ever would have spent if we’d tried to keep the key technologies closer to home.”
The right goal: add value for customers
Let’s start with what Boeing did right. After losing market share to Airbus (owned by EADS) in the late 1990s, Boeing could have decided to focus on reducing the costs (and the selling prices) of its existing aircraft. That would have led inexorably to corporate death. Instead Boeing decided—commendably—to innovate with a new aircraft that would generate revenues by creating value for customers.
First, Boeing aimed to improve their travel experience for the ultimate customers, the passengers. As compared to the traditional material (aluminum) used in airplane manufacturing, the composite material to be used in the 787 (carbon fiber, aluminum and titanium) would allow for increased humidity and pressure to be maintained in the passenger cabin, offering substantial improvement to the flying experience. The lightweight composite materials would enable the 787 to fly nonstop between any pair of cities without layovers.
Second, Boeing aimed to improve value for its immediate customers (the airlines) by improved efficiency by using composite materials and an electrical system using lithium-ion batteries. This would result 20 percent less fuel for comparable flights and cost-per-seat mile 10 percent lower than for any other aircraft. Moreover, unlike the traditional aluminum fuselages that tend to fatigue, the 787′s fuselages based on composite materials would reduce airlines’ maintenance and replacement costs.
All good stuff, if Boeing could deliver. Boeing’s customers apparently thought they could. And the 787 became the fastest selling plane in aviation history. The stock price popped and the C-suite received their bonuses. But reality has since set in.
Overheating batteries
We have no way of knowing whether the cause of the current grounding of all 787s—lithium-ion batteries that overheat alarmingly—is a narrow, fixable manufacturing glitch or a serious design flaw that will put the whole enterprise in peril.
It’s true, as CEO James McNerny pointed out in a letter to Boeing staff on Friday, that “Since entering service 15 months ago, the 787 fleet has completed 18,000 flights and 50,000 flight hours with eight airlines, carrying more than 1,000,000 passengers safely to destinations around the world.” But all that will mean nothing unless and until Boeing can get to the root cause of those overheating Lithium-ion batteries.
What we do know is that the cost-cutting way that Boeing went about outsourcing both in the US and beyond did not include steps to mitigate or eliminate the predicted costs and risks that have already materialized.
The coordination risk
Even with proven technology, there are major risks in outsourcing that components won’t fit together when the plane is being assembled. “In order to minimize these potential problems,” wrote Dr. L. J. Hart-Smith, a Boeing aerospace engineer, in a brilliant paper presented at a 2001 conference, “it is necessary for the prime contractor to provide on-site quality, supplier-management, and sometimes technical support. If this is not done, the performance of the prime manufacturer can never exceed the capabilities of the least proficient of the suppliers. These costs do not vanish merely because the work itself is out-of-sight.”
Boeing did not plan to provide for such on-site support for its suppliers. In fact, it explicitly delegated this responsibility to sub-contractors. When the subcontractors didn’t perform the necessary coordination, Boeing had to provide the support anyway. “Boeing sent hundreds of its engineers to the sites of various Tier-1, Tier-2, or Tier-3 suppliers worldwide to solve various technical problems that appeared to be the root cause of the delay in the 787′s development. Ultimately, Boeing had to redesign the entire aircraft sub-assembly process.” The result? Huge additional expense, that should have been planned for and included in the project’s costs from the outset.
The innovation risk
The 787 involved not merely the outsourcing of a known technology. It involved major technological innovations unproven in any airplane. Would the carbon fiber composite survive the rigors of international flying? Could lithium-ion batteries, which are notorious for overheating and causing fires that are difficult to put out, be safely used? No one knew for sure. The 787 also contains multiple new electrical systems, power and distribution panels. The interactions among these novel technologies, introduced simultaneously, also exponentially increased the risk of innovation.
The innovation risk implied a greater involvement by Boeing in the development and manufacture of the aircraft. Astonishingly, Boeing opted for lesser involvement, delegating much of the detailed engineering and procurement to sub-contractors. The result? Unexpected problems have kept occurring that have delayed the project and increased its cost.
The outsourcing risk
Complicated products like aircraft involve a necessary degree of outsourcing, simply because the firm lacks the necessary expertise in some areas, e.g. engines and avionics. However Boeing significantly increased the amount of outsourcing for the 787 over earlier planes. For the 737 and 747 it had been at around 35-50 percent. For the 787, Boeing planned to increase outsourcing to 70 percent.
Boeing didn’t approach outsourcing as a troublesome necessity. Instead, like many US firms, it enthusiastically embraced outsourcing in the 787 as a means of reducing costs and the time of development. “The 787′s supply chain was envisioned to keep manufacturing and assembly costs low, while spreading the financial risks of development to Boeing’s suppliers.”
In his 2001 paper, Hart-Smith had warned of the additional costs and risks of large-scale outsourcing. Outsourcing didn’t cut costs and increase profits, he wrote; instead, it drove profits and knowledge to suppliers while increasing costs for the mother company. “Not only is the work out-sourced; all of the profits associated with the work are out-sourced, too.”
Hart-Smith argued that make-buy decisions should be based on complete assessments of all of the costs: “make-buy decisions should not be made until after the product has been defined and the relative costs established.” Outsourcing requires considerable additional up-front effort in planning to avoid the situation whereby major sub-assemblies do not fit together at final assembly, increasing the cost by orders of magnitude more than was saved by designing in isolation from the work-allocation activities.
Boeing didn’t follow Hart-Smith’s advice and outsourced the engineering and construction of the plane long before the product was defined and the relative costs established. The results have been disastrous. Boeing’s 787 project is many billions of dollars over budget. The delivery schedule has been pushed back at least 7 times. The first planes were delivered over three years late.

con't...

[Jeff Higgins]

...con't

The risk of tiered outsourcing
Boeing further aggravated these risks by adopting a new outsourcing model, along with the new technology. Unlike Boeing’s earlier aircraft, in which Boeing played the traditional role of integrating and assembling different parts and subsystems produced by its suppliers, the 787′s supply chain is based on a tiered structure that would allow Boeing to foster partnerships with around fifty Tier-1 strategic partners. These strategic partners were to serve as “integrators” who assemble different parts and subsystems produced by Tier-2 and Tier-3 suppliers.
In due course, Boeing discovered, as Hart-Smith had predicted, that some Tier-1 strategic partners did not have the know-how to develop different sections of the aircraft or the experience to manage their Tier-2 suppliers. To regain control of the development process, Boeing was forced to buy one of the key Tier-1 suppliers (Vought Aircraft Industries) and supply expertise to other suppliers. Boeing also had to pay strategic partners compensation for potential profit losses stemming from the delays in production.
The risk of partially implementing the Toyota model
Boeing’s outsourcing was modeled in part on Toyota’s supply chain, which has enabled Toyota to develop new cars with shorter development cycle times. Toyota successfully outsources around 70 percent of its vehicles to a trusted group of partner firms.
However key elements of the Toyota outsourcing model were not implemented at Boeing. Toyota maintains tight control over the overall design and engineering of its vehicles and only outsources to suppliers who have proven their ability to deliver with the required timeliness, quality, cost reduction and continuous innovation. As Toyota works closely with its suppliers and responds to supplier concerns with integrity and mutual respect, it has established an impressive level of professional trust and an overriding preoccupation with product quality.
By contrast, Boeing adopted the superficial structure of Toyota’s tiered outsourcing model without the values and practices on which it rests. Instead, Boeing relied on poorly designed contractual arrangements, which created perverse incentives to work at the speed of the slowest supplier, by providing penalties for delay but no rewards for timely delivery.
The offshoring risk
Some degree of outsourcing in other countries—i.e. offshoring—is an inevitable aspect of manufacturing a complex product like an airplane, because some expertise exists only in foreign countries. For example, the capacity to manufacture Lithium-ion batteries lies outside the US. Boeing had no choice but to have the batteries made in another country. More than 30 percent of the 787’s components came from overseas. By contrast, just 5 percent of the parts of the 747, were foreign-made.
While there is nothing in principle wrong with necessary offshoring, the cultural and language differences and the physical distances involved in a lengthy supply chain create additional risks. Mitigating them requires substantial and continuing communications with the suppliers and on-site involvement, thereby generating additional cost. Boeing didn’t plan for such communications or involvement, and so incurred additional risk that materialized.
The risk of communications by computer
Rather than plan for face-to-face communications and on-site communcations, Boeing introduced a web-based communications tool called Exostar in which suppliers were supposed to input up-to-date information about the progress of their work. The tool was meant to provide supply chain visibility, improve control and integration of critical business processes, and reduce development time and cost. Instead of people communicating with people face-to-face, the computer itself was supposed to flag problems in real time.
Not surprisingly, the tool failed. Suppliers did not input accurate and timely information, in part due to cultural differences and lack of trust. As a result, neither Tier-1 suppliers nor Boeing became aware of problems in a timely fashion. Boeing’s reliance on computer communications contrasts sharply with Agile practices of continuous face-to-face communications to ensure that everyone is on the same page.
The labor relations risk
We do not know to what extent Boeing’s enthusiasm for outsourcing and offshoring stemmed from a desire to circumvent difficult labor relations in Seattle. We do know that instead of involving the employees in the decision-making about outsourcing and offshoring, Boeing’s management approached decision-making pre-emptively. The approach backfired, as labor relations worsened as a result of the outsourcing decisions and a costly strike ensued.
The project management skills risk
Given the extraordinary risks of the 787 project, one would have expected Boeing to assemble a leadership team with a proven record in supply chain management and diverse expertise to anticipate and mitigate wide array of risks. Amazingly, this was not the case.
“Boeing’s original leadership team for the 787 program,” write Tang and Zimmerman in an important case study, “did not include members with expertise on supply chain risk management. Without the requisite skills to manage an unconventional supply chain, Boeing was undertaking a huge managerial risk in uncharted waters.”
The risk of a disengaged C-suite
The combination of the above risks constituted an existential threat to Boeing as a going concern. Where then was the C-suite while these risks were being incurred? An interview in 2011 with Philip Condit, who was the richly compensated CEO of Boeing when the initial 787 decisions were being made, is revealing.
In 2001, under Condit’s leadership, Boeing moved its headquarters from Seattle to Chicago, a decision continued by Condit’s successor, James McNerney. The ostensible reason for the move was to be neutral among the various divisions of Boeing, which were scattered around the US. In the interview, Condit makes no secret of another factor: as CEO, he didn’t want to be bothered with tiresome “how-do-you-design-an-airplane stuff,” or boring meetings with Boeing’s key customers (airlines) who came to Seattle.
After the move, Condit says that he spent much of his time in the Chicago business community, where he “encountered CEOs frequently gathering to nail down civic goals ranging from landing new companies to building world-class parks. ‘I was surprised by how much that happened,’ Condit said. ‘A meeting in which Starbucks, Microsoft, Costco, Boeing and Weyerhaeuser and a bunch of small businesses are all in the same place — rarely happens in Seattle,” he added. ‘It happened all the time in Chicago.’”
So while Boeing’s CEO was in Chicago, strategizing about the future of Boeing and discussing civic goals with CEOs from other companies, the managers back in Seattle were making business decisions about tiresome “how-do-you-design-an-airplane stuff” that would determine whether there would be a firm to strategize about.

[onewhippedpuppy]

Excellent article Jeff. There are numerous examples of failed outsourcing, and the 787 program is very similar to a jet program at my employer that was a significant commercial failure. Companies need to learn from these failures to temper the enthusiasm for oursourcing, which often has a "forbidden fruit" element to it. The numbers are so tempting on paper that it is difficult to resist, yet the numbers on paper are very rarely achieved.

[IROC]

Originally Posted by onewhippedpuppy

Very difficult to coordinate suppliers around the world that are responsible for complex and interconnected parts.

This is exactly what Boeing claims is its strength, though. They claim to be a "large scale systems integrator". Look at ISS and GMD. That's exactly what those programs required.

One instance in which outsourcing typically is not worth the trouble, at a minimum Boeing should have done the design work in-house and outsourced manufacturing (IMHO).

Boeing saw outsourcing as a way to increase "shareholder value". I think the pendulum swung too far, though. Boeing decided that they could make more money paying someone else to do the design and fabrication and simply "manage the interfaces" and do the final assembly. When I was a design engineer on the GMD program, all of design engineering (and that was large program) was brought into a room where we were told we were "dinosaurs" and that if we wanted a future at Boeing, we needed to change our skill set to become "system engineers".

Boeing is loaded with sharp people and problems like this are simply going to happen. Boeing needs to work through the problem and solve it. That's all.

[scottmandue]

[cairns]

Thank you for that Jeff. As one who works frequently with Boeing as a subcontractor (but not on the 787) I will say that Boeing needs to update it's supplier quality systems and that their use of Exostar on the 787 is a big mistake. While Raytheon, Lockheed, and many other firms like them also use it, IMO it's a dated PITA that should never be used on complex systems.

As to their supplier quality systems I am still receiving, every week, Boeing quality and deliquency reports on contracts that ended years ago.

Given all this I'm still a big Boeing fan. I really hope they can find the flaw in the batteries quickly and get those planes back in the air.

[stevepaa]

Good article Jeff.
We had an issue arise two years ago at a tier 2 which I thought would never happen because the engineering manager came from us. But he did not control quality. We are just now recovering and we have had the tier 1 sub put someone at the tier 2 to watch over them.

[Rusty Heap]

Having worked on the composite bodys electrical grounding wiring, that plane is the most well grounded same electrical/static noise floor for any component anywhere in the plane. It's not composite rubbing on each other.

The electrical system is also one of the most advanced and highest power ever built.

There are six starter generators on each Boeing 787: two auxiliary starter generators on the Hamilton Sundstrand-supplied Auxiliary Power Unit (APU), and two variable frequency starter generators (VFSGs) mounted on each Trent 1000 main engine.
An integral part of Hamilton Sundstrand's 787 Electric Power Generating and Start System (EPGSS), the generators collectively supply nearly 1.5 megawatts of power, which is enough to power about 400 homes, and more than five times the electric power on a current Boeing 767. Combined with the elimination of the high pressure pneumatic system, this results in an overall reduction of power required at cruise.

PR Newswire (http://s.tt/1sMzm)


Taking a Peek into Boeing 787 Electrical Power System | The Skyward Blog

AERO - Boeing 787 from the Ground Up

[5String43]

Plus this:

Boeing's troubles could be traced to McDonnell Douglas purchase

By Mark Lacter | January 29, 2013 9:51 AM

With its 787 Dreamliner still grounded after battery fires on two planes, and no sign of when the wide-body aircraft will be back in the air, Boeing is under fire on many fronts. Too much outsourcing, too little testing, too great a technology risk - it's a full-fledged corporate nightmare. But to fully understand what happened, it might be helpful to revisit the 1997 acquisition with McDonnell Douglas (based in St. Louis but with a major operation in Long Beach). Seems as if the bean counters took control of the engineers - never a good thing. From New Yorker columnist James Surowiecki:

Technically, Boeing bought McDonnell Douglas. But, as Richard Aboulafia, a noted industry analyst with the Teal Group, told me, "McDonnell Douglas in effect acquired Boeing with Boeing's money." McDonnell Douglas executives became key players in the new company, and the McDonnell Douglas culture, averse to risk and obsessed with cost-cutting, weakened Boeing's historical commitment to making big investments in new products. Aboulafia says, "After the merger, there was a real battle over the future of the company, between the engineers and the finance and sales guys." The nerds may have been running the show in Silicon Valley, but at Boeing they were increasingly marginalized by the bean counters.

Under these conditions, getting the company to commit to a major project like the Dreamliner took some doing. "Some of the board of directors would rather have spent money on a walk-in humidor for shareholders than on a new plane," Aboulafia says. So the Dreamliner's advocates came up with a development strategy that was supposed to be cheaper and quicker than the traditional approach: outsourcing. And Boeing didn't outsource just the manufacturing of parts; it turned over the design, the engineering, and the manufacture of entire sections of the plane to some fifty "strategic partners." Boeing itself ended up building less than forty per cent of the plane. This strategy was trumpeted as a reinvention of manufacturing. But while the finance guys loved it--since it meant that Boeing had to put up less money--it was a huge headache for the engineers.

[IROC]

Originally Posted by 5String43

Plus this:

Boeing's troubles could be traced to McDonnell Douglas purchase

As a "heritage McDonnell Douglas" guy, I can say that there is a difference of opinion on this subject within current Boeing employees...

I was at one of the few McDonnell Douglas/Boeing locations where immediately after the "merger" employees from both companies began working side-by-side. McDonnell Douglas was doomed by a lack of cash reserves and a shrinking market share. Boeing was being run by ludicrously high-paid idiots - at least at our location. Boeing engineers were making about 30% more than equivalent McDonnell Douglas guys and were so hated by local NASA management that many McDonnell Douglas guys were moved into management positions on Boeing contracts right away.

[Jeff Higgins]

Originally Posted by IROC

As a "heritage McDonnell Douglas" guy, I can say that there is a difference of opinion on this subject within current Boeing employees...

Yes, there is. The former Mickie-Dee's employees have a grossly overestimated view of themselves, particularly their "engineers" (who in reality were no more than "project managers") by then. Everyone else in the industry agrees - the problems at Boeing are a direct result of adopting MD management philosophies and managers, wherein engineers quit designing and picked up project managing instead. Google "adopting the practices of a failed aerospace company" (or something like that) for some enlightening reading.

Originally Posted by IROC

I was at one of the few McDonnell Douglas/Boeing locations where immediately after the "merger" employees from both companies began working side-by-side. McDonnell Douglas was doomed by a lack of cash reserves and a shrinking market share.

A lack of cash flow and dwindling (actually non-existant by then) market share brought about by the extreme risk-adverse culture that Aboulafia mentions. No one's fault but MD's - their corporate culture led directly to that sad state of affairs. Every industry analyst that has ever bothered to comment on that situation is in agreement.

Hell, by that time, McDonnel had taken one previously proud, productive, civilian sector manufacturer - Douglas - and ruined it with his miserly, government contract dependent, penny pinching ways. He would never invest his own money - only taxpayers' money. That's what killed MD as a viable commercial airplane manufacurer. That, and trying to use its remaining "engineers" as project managers instead.

Originally Posted by IROC

Boeing was being run by ludicrously high-paid idiots - at least at our location. Boeing engineers were making about 30% more than equivalent McDonnell Douglas guys and were so hated by local NASA management that many McDonnell Douglas guys were moved into management positions on Boeing contracts right away.

Actually, by then, there were no "equivalent MD guys" - whoever was left there as an "engineer" had been turned into a project manager. All the guys who really wanted to be (and had the werewithal to be) design engineers had already left.

So of course those who were left saw the Boeing engineers as "idiots" - they couldn't understand what they were doing, why it took so long, and why it costs so much. That's a very (all too) common view of engineering work from the outside looking in; and a view that got MD into the predicament it was in then, and subsequently allowed MD management to force Boeing into its current situation.

And that's where the failure began, as chronicled by many industry experts - when Mickie-Dee's management was given charge over Boeing engineers. The tail wagging the dog in the most engineering-intensive undertaking mankind has ever tackled. We all knew it was wrong, and where it would lead.

MD had a long, sordid history of undervaluing engineering. They saw it as just another commodity, that could be purchased off the shelves of a world market. Maybe the "talent" they had at the time led them to that conclusion, or maybe that attitude led to the "talent" they were left with - a real "chicken or the egg" question. But I digress...

They were the first to significantly outsource engineering. We simply never did that at Boeing, until MD bean counters came in. Prior to that, Boeing had always valued the engineering culture that was developed here, and understood that it could not be matched anywhere. That culture has always attracted the cream of the crop, because engineers knew they would be working not only with other engineers, but more importantly, for other engineers. The rest wound up at places like, well, MD - where they didn't "engineer" a damn thing, but turned into "project managers" riding herd over incompetent suppliers, wondering why engineers are such "idiots"...

[onewhippedpuppy]

Originally Posted by Jeff Higgins

So of course those who were left saw the Boeing engineers as "idiots" - they couldn't understand what they were doing, why it took so long, and why it costs so much. That's a very (all too) common view of engineering work from the outside looking in; and a view that got MD into the predicament it was in then, and subsequently allowed MD management to force Boeing into its current situation.

And that's where the failure began, as chronicled by many industry experts - when Mickie-Dee's management was given charge over Boeing engineers. The tail wagging the dog in the most engineering-intensive undertaking mankind has ever tackled. We all knew it was wrong, and where it would lead.

MD had a long, sordid history of undervaluing engineering. They saw it as just another commodity, that could be purchased off the shelves of a world market. Maybe the "talent" they had at the time led them to that conclusion, or maybe that attitude led to the "talent" they were left with - a real "chicken or the egg" question. But I digress...

They were the first to significantly outsource engineering. We simply never did that at Boeing, until MD bean counters came in. Prior to that, Boeing had always valued the engineering culture that was developed here, and understood that it could not be matched anywhere. That culture has always attracted the cream of the crop, because engineers knew they would be working not only with other engineers, but more importantly, for other engineers. The rest wound up at places like, well, MD - where they didn't "engineer" a damn thing, but turned into "project managers" riding herd over incompetent suppliers, wondering why engineers are such "idiots"...

I have no first hand knowledge of the Boeing/MD merger, but wow does that resonate with me. Change the name and that describes the cultural change of my employer.....which is currently in bankruptcy.